GroupVitals System Status
If there are any major issues with our system, we'll post them here 👍
All Good
Email Deliverability Issues
February 17th, 2022
TL;DR ⏲
Gmail, in a very unusual and temporary move blocked/bounced a bunch of emails being sent via groupvitals.com on Feb 15-16th and our email provider, MailChimp added all those blocked/bounced emails to their rejection list. If someone has stopped receiving emails from groupvitals.com starting on February 15th, they're likely on the rejection list. We can remove them from this list after February 24th, 2022, when we launch an update that improves how we handle blocked/bounced emails.
👉 Please fill out this form to get an email removed from the rejection list after February 24th, 2022.
Long Version 👇
We had an unusual email event with Gmail email addresses on February 15th and 16th, 2022. On these two days, Gmail/Google decided they weren't liking a lot of the emails being sent from groupvitals.com, so they started "bouncing" them back to us / rejecting them.
Once this event started, the email system GroupVitals uses (MailChimp Enterprise) quickly added these Gmail accounts to the rejection list.
If you or others are experiencing not receiving any emails from GroupVitals starting February 15th, they're likely affected from this event and their email has been added to MailChimp's rejection list for groupvitals.com
We'll be able to remove affected emails from the rejection list, but first, we need to improve how our system handles emails that get bounced and rejected, to prevent emails from being added back to the rejection list if they get bounced again. We're aiming to have this complete by February 24th, 2022 🤞
Removing emails from the rejection list has negative consequences for GroupVitals.com email reputation. The more emails we remove from the rejection list, the more likely MailChimp and Google will not send out emails on behalf of groupvitals.com. So we're proceeding with an abundance of caution so that you and others can more safely and reliably receive emails from groupvitals.com.
Please fill out this form to get an email removed from the rejection list after February 24th, 2022.
We apologize for the temporary disruption in email service to those who have been impacted and thanks for your patience here 🙏
If you're within the United States, consider setting up text alerts for attendance and GroupFinder notifications.
Past Status Updates
🏁 FellowshipOne Issue
Last Updated on 12/18/20 @ 4:00 PM CT
We've updated everyone's FellowshipOne accounts with missing groups, group updates and roster changes that were not originally sent due to F1's bug.
F1 fixed their bug with the API and our internal testing shows all group updates are working again 🙌
Now we're working on sending all the group updates that have been made since 12/15 that were never sent to F1. Hopefully, we'll get that all done before Monday.
TL:DR
GroupVitals can't send group updates (info and roster) to F1 right now and we've disabled the nightly sync until we've resolved this issue. After looking at all of our recent code changes, we believe it's a bug from F1's recent product update and their team is currently investigating the issue and we're waiting to hear from them.
Important Things to Know
- All group updates can still be made in GroupVitals, they just won't be sent to F1 until we get this issue resolved (as we keep a log of all updates to be sent to F1).
- We've disabled the nightly sync with F1 until this issue has been resolved. This will prevent a lot of group data in GroupVitals from getting reset in GroupVitals by the nightly sync. This also means updated people records and new people records will not be sent to GroupVitals until we turn this back on.
- Group attendance is still working and being sent to F1.
- New people and updates to people in GroupVitals continue to be sent to F1 without any problems.
- We'll keep this page updated with the most up-to-date information.
🏁 Attendance Issue
4/15/20 @ 11:30 AM CT
Attendance is a bit backed up this week. Rest assured, it's been recorded, but not all have been processed yet. We hope to have them processed over the next 12-24 hours.
We're working on getting all attendance processed, then we'll work on ways to keep it from getting backed up like this again.
🏁 Email Downtime
1/11/19 @ 6 AM CT
Our email system is back up and all emails that were supposed to go about between midnight and 6am have been sent
1/7/19 @ 11 AM CT
We wanted to let you know that GroupVitals' email system will be down for maintenance this Saturday, January 11th, 2020. You can still login to GroupVitals and send emails on Saturday. They just won't be sent to the recipients until the system is back online.
We'll do our best to get every automated and manual email sent that day to go out as soon as the email system is back up, or Sunday at the latest. If you have planned on sending any emails Saturday, you may want to send them before then.
FYI - We use MailChimp's enterprise Mandrill service to send emails with GroupVitals. Mandrill recently informed us that they are having this downtime to make system updates. This downtime is out of our control and something they've never done before with us. We're doing our best to roll with it and make some updates to our software so that your Saturday emails will get sent out as soon as Mandrill gets their email system back online.
🏁 Attendance Issue
5/16/19 @ 5 PM CT
At this point, all but one church has finished re-syncing data between GroupVitals and F1. Any missing attendance in F1 should also be showing up in F1 now. If you're seeing anything that isn't syncing or matching up, please contact me - [email protected] and we'll get it all worked out 🙏I'm sorry for the troubles here and I thank you guys for your patience! - Tony
5/15/19 @ 9:30 AM CT
We're about 90% caught up with syncing all accounts with F1. We have just a few accounts with very large databases that we're pulling in updated data from their F1 account. We expect this to be done by the end of the day. After that, 100% of accounts will be back in full operation with our F1 sync 👍
5/14/19 @ 12:30 PM CT
We're just about done with getting F1 back to syncing with GroupVitals. We've pushed over to F1 all the changes made in GroupVitals. Now, we're pulling over all the updates that had been made to people and groups while our integration with F1 was down. We expect this to be done by the end of today. After that, everything should be in sync and the nightly F1 sync will be turned back on and everything full operational.
5/13/19 @ 2:30 PM CT
We've made a lot of progress over the weekend getting all of the F1 <> GroupVitals sync data in order. We've also had to write some custom code/scripts in order to complete this project of updating F1 accounts with the correct data. We're still in the process of comparing records to see which one is the correct one to update in either F1 or GroupVitals. Once that's done, we'll be able to begin the process of updating accounts. At this rate, we're hopeful we'll be all done by this Tuesday.
5/10/19 @ 10 AM CT
I wanted to let you know that we're currently experiencing some issues with FellowshipOne. The nightly sync is currently down, so any changes made in F1 are not being updated in GroupVitals right now. However, any changes made in GroupVitals are currently being sent to F1. We're working hard to getting the nightly sync back up and running by May 13th or 14th.
What happened:
On May 9th, we discovered that between April 20th and May 9th, any changes made in GroupVitals were not sent to F1. For some churches, it started on April 20th and for others, it started on May 9th, which is what made this issue hard to catch. The great news is that we have a record of all the updates that were not sent to your F1 account during this down period. The reason why the sync stopped working is that each month, F1 requires us to renew a special "token" for each church (kinda like a password) in order to keep syncing data with the church. These tokens expired and were not automatically renewed by our system as usual. There was an error in our code that prevented the token renewing from happening (that's been fixed now). So once a token expires, updates made in GroupVitals or F1 are prevented from transferring from one to the other. We've already renewed every church's token, so that's how you can currently make changes in GroupVitals and those will immediately be sent to F1.
What we're doing:
While the nightly sync with F1 is paused, we're going through our logs for each church to determine when their sync stopped, what information needs to be updated in F1 and what information needs to be updated in GroupVitals (based upon changes made in F1). Once we determine this information, we'll then proceed to send either send updates made to people and groups to F1 or pull updates from F1 to GroupVitals. It will all depend upon where and when the information was last updated for a person or group. Once we get all the data updated and in sync, we'll be able to flip on the switch for the nightly sync with F1.
What you should do:
Please avoid performing a manual sync with F1 via the add-ons section (it won't really work). If possible, avoid updating anything related to a group within F1. We know that people will be updated within F1, so you/other staff can continue updating and adding people records in F1 - we'll bring over all those updates once we sort through the data and launch our fix.
I'll continue to post updates to this page.
If you have any questions, don't hesitate to reach out: [email protected]
🏁 Attendance Issue
Update 2/20/19 @ 8:20 AM CT
Attendance is now sending out normally. We've added some internal tracking to know timestamp exactly when each attendance reminder was sent out for each group. We're also going to setup a couple more instances throughout the day where we'll double check that everyone who was supposed to receive the attendance reminder, in fact, received it. If not, it will try re-sending again. Sometimes our email provider (MailChimp's Mandrill) gives us an error when sending out emails and it will stop sending all in that batch after it gets that error, preventing emails from going on. So by adding a few more checks throughout the day, we'll be able to help ensure that emails that didn't get sent before will get sent.
2/19/10 @ 10:00 AM CT
There's a small percentage of groups that aren't getting their attendance notifications. We're digging in to see why that is.
🏁 Notifications Issue
Update 11/5/18 @ 1:35 PM CT
We got it! We discovered that there was just one affected email notification setting that wasn't working properly. It was the "attendance has been taken" email notification. If you had this notification OFF before our update last week, it was actually being sent out. This primarily affected account admins/staff. If someone was a group leader or member and receiving these notifications, it's because they are also a staff/admin on the account. We've fixed the bug and anyone who had the "attendance has been taken" notification toggled to OFF will no longer receive these notification emails with our fix. We're so sorry for flooding some inboxes with these notifications over the weekend.
Lastly, we fixed the issue with the My Notifications page not saving changes. So you're welcome to head over there and double check your notifications to be sure everything is set the way you currently want it.
Update 11/5/18 @ 11:50 AM CT We've received reports that email notifications that were toggled to OFF are being sent out and saving some notification settings isn't working. We're currently working on getting this fixed asap.
🏁 Attendance Issue
Update 10/22/18 @ 4:00 PM CT
We found and fixed a bug with our attendance processing. Most groups attendance was submitted that was taken while the bug was in place. However, some groups may have not had their attendance saved/processed. They'll have another chance to do that tomorrow when our attendance not taken yet notices go out.
Update 10/22/18 @ 3:30 PM CT We've received some reports that attendance isn't saving for some groups. We're currently looking into this.
🏁 Slow Performance
Update 10/5/18 @ 7:30 AM CT
We're back and faster than ever 🔥By now, you should be noticing how fast everything is. We've moved over to Google's Cloud Platform, which has sped things up considerably and now we have unlimited resources to grow and expand because of Google's excellent technology and infrastructure. Thanks again for your patience this week. If there are any issues you run into, please don't hesitate to contact us.
Update 10/4/18 @ 10:25 AM CT
We're about 85% ready for moving everything to Google Cloud. We're doing lots of testing to ensure everything is fast and functional. If everything goes as planned, we'll switch GroupVitals over to Google tomorrow morning.
Update 10/3/18 @ 10:55 AM CT
We're making progress with moving everything to Google Cloud 👍We've got the basic application up and running and now we're just fine tuning things. I personally just tested it and everything is fast.. faster than it was before 🎉We've run into a snag with our F1 API that we are trying to fix at the same time as getting everything moved over to Google. If we don't make the move Thursday morning, we'll shoot for Friday morning. Thanks again for your patience!
Update - 10/2/18 @ 1:30 PM CT
We're still on track to move the entire GroupVitals application to Google Cloud this week. The goal is to move it early Thursday morning. Thanks again for your patience with the slower system this week
Update - 10/1/18 @ 11 AM CT
In order to solve the slowness of the system, we'll be continuing ahead with our plan to move the application to Google's cloud platform. We planned to do within the next few weeks, but we're going to make the move this week. Getting both the database and the application hosted with Google in the same environment will increase the speed and allow us to expand our resources as needed. Thanks for your patience and we'll keep you updated on this page!
Update - 9/28/18 @ 8 AM CT
We've moved our database to Google Cloud early this morning. We're noticing a slower performance in loading pages and saving changes and we're currently working on improving that. We'll update this page when we've got things loading faster.
🏁 Database Move
Update - 9/29/18 @ 8:00 AM CT
The database has been successfully moved over to Google Cloud 🙌
Update - 9/28/18 @ 12:00 AM CT
We've started the process of moving our database from a private server to Google Cloud in order for us to better support future growth and better performance. This is the first phase of moving our entire system to Google's Cloud over the next month.