Roaming Children?

There are a few young boys (ages 6 & 7) that are always out and about with no parents to be seen. One day a couple weeks ago they made friends with Logan (and CJ who likes to butt in lol) when we were all out playing with bubbles and chalk. Since then they have come by a few times wanting to play. I feel kind of bad because I always say no. But they come in the evening on school days and I am just not up to going outside with all the boys when I'm needing to do dinner and clean up and tend to the baby. And above all, I am NOT going to let MY boys go outside by themselves. I have considered inviting them to play in the backyard at times (that way I know my boys are safe and contained but they still get to play with their friends) but I don't want to have these kids back there with out knowing that their parents know where they are so they can find them if they decide they want to. I know I played outside when I was little, but 6 just seems really young to me. Logan asked today, after the boys left when he told them he couldn't play, "why don't they ask me to play on the weekend?" I guess he isn't as upset about not getting to play because he knows it's a school night and that we have a routine on school nights. He's such a good boy!