I know that Open edX has Google Hangouts integration via the Instant Hangouts plugin

http://edx-partner-course-staff.readthedocs.org/en/latest/exercises_tools/google_hangouts.html

MIT is has also built a service called Unhangout that might be more flexible and suitable to your needs. It’s also open source which means we could probably integrate it with Open edX without too much effort.https://unhangout.media.mit.edu