Jeez, I think you are worrying too much.
My only experience is mountain-biking. We used to do group rides every Saturday. For the first couple of years I was all about wanting to speed through it, cover as much ground as possible, and leave the slower people behind. I saw my rec time as being precious, and I wanted to improve my fitness.
Now I am embarrassed about those days. I had joined a group ride, but I was still thinking of myself.
So I think you find out what the purpose of the ride is. Is it a group ride, or is there some other goal. e.g training for an event, setting a new record, or visiting some place and being home before dark. If it's a group riding together for fun, I'm sure the fast guys are enjoying the chat time just as much as the ride time.