The Short Answer:
Guinea pigs are fine with eating all types of peppers including bell peppers. The only thing you need to be sure of is to not overfeed them, and to remove all the seeds, and the stem of the pepper. The seeds are highly toxic, and the stem is not suitable for their diet.
Bell peppers are spicy, so you should not give them a large amount. They are high in vitamin C, so they are healthy, but in high amounts it can have negative effects on your guinea pigs stomach, and digestive system. You should only give them a small amount.
The First Time:
Like all animals there is a chance that your guinea pig will be allergic to the bell pepper. It is best to keep an eye on the guinea pig when you give it the first piece (a small amount). Because it is spicy, your guinea pig may act a little strangely, but watch for signs of trouble breathing, and elongated strange behavior.