System Status‎ > ‎

Status Archive

--12/12/13--
Enrollment Manager Outage 1:45PM-2:00PM The data center team was working to remove a hard drive from one of our database clusters. The drive was no longer used and had no data. The team had successfully decommissioned the drive, but when actually removing it caused a monitoring system for the cluster to alert and in response turn off access to the databases. The team realized their problem and took action to restore the hard drive.

--11/14/13--
Overnight workflow processing stopped. Processing was restarted but some workflows were delayed .

--10/14/13--
Imports were delayed over the weekend due to a problem affecting the server. This was corrected and all queued imports were finished by 1PM EDT 10/14

--8/27/13 -- 
An issue caused users to be unable to save Person records. This also affected merging person records.

--7/30/2013--

A hardware issue with one of our servers caused a loss of functionality for a number of Enrollment Manager features beginning early in the morning on 7/30. Our data center engineers found the problem and restored functionality by 10:30AM EDT.

--7/26/2013--

An issue with our email provider caused a security token error to appear on mailing records for some clients. This was fully resolved by 2:30PM

--5/21/2013--

An issue was discovered that caused the "Get Merge Data," "Send Bulk Email," and "Post Activities" buttons to not load on an Outreach List record. It also caused the window to freeze. A fix was applied by 11:00 AM to all organizations.

--5/15/2013-- Issues from the May Update
  • The Standard Opportunity and Standard Interest Reports are experiencing an error when you try to run them.We are working to roll out a fix. If you experience an error, please contact support and we can assist with running the report.
  • Error occurring on the Person record affected users with the SSN role  and caused the State and Country field to appear blank.
  • The migration of records from Relationships to Connections completed Monday around 12 for all organizations. In the process, some duplicates were created but later cleaned up. Also, some connections were created without roles. These roles have been back filled and should appear correct now.
  • The CRM for Outlook plugin began performing slowly after the update. If you are experiencing problems, please try opening the CRM Configuration Wizard, deleting the organization, and then adding your organization back.

--5/3/2013--

A problem on one front end web server caused a problem with logging in to Enrollment Manager. This began early in the morning and was resolved at 9:15 AM EST.

--4/29/2013--

A problem with our email provider caused errors with the Quick Mailing function beginning early in the morning. Mailing definitions would not auto-populate when chosen and manually filling in the details also produced an error. Developers worked on the problem throughout the day and had a fix in place by 7:00 PM Eastern. 

-- 4/10/2013 --

Overnight workflow processing was delayed due to higher that normal volume last night. All workflows continued to process and were caught up by 12:00 PM.

9:45 AM to 10:15 AM
An error with Web Forms caused the site to become inaccessible. 

-- 3/28/2013 --

Our email provider had an error that occurs when you try to send a mailing from an outreach list using the "Send Bulk Email." This also affected quick mailings. The error was officially resolved by 2:00PM

--3/11/2013--

A problem with our email provider was causing a "Security Token" error in Mailing records or when trying to send a bulk email from an Outreach List. The issue has now been corrected.

--3/8/2013--

Our email provider is reporting a problem with their system. This is affecting sending a bulk email from an Outreach List (Using the Send Bulk Email button). It will also affect sending a tracked email. Sending bulk email from a Mailing record should not be affected. Our provider has corrected the issue and bulk email should be functioning as normal.


3:30PM A problem with one web server was causing new logins to fail. We corrected the problem and all access was restored by 3:45PM.

--2/27/2013--
An problem began early in the morning with one of our Enrollment Manager servers that caused an error message to display when trying to edit a Mailing record or send a bulk email. After investigating, the trouble server was identified at 11:50AM and removed so it could not cause the problem while it was repaired. 

--2/4/2013--
A connection issue with our email provider caused a "Security token" error in the HTML area of a Mailing record beginning around 3:30PM EST. This was resolved by 4:30PM EST.

--1/31/2013 --
A database issue reported at 7PM EST was causing slow performance within Enrollment Manager. The issue was resolved and performance back to full speed at 8:15 PM EST. 

-- 11/15/2012 --
Overnight, workflow volume began affecting the disk activity of the database server. The backlog caused problems navigating in Enrollment Manager and performing certain functions. Our data team discovered the process that was starting a high volume of bad workflows and stopped them by 1:30PM.

-- 10/22/2012 --
An issue began around 6:00AM that caused some users to receive a 404 error after entering their credentials. Our server team investigated and resolved the issue by 9:10AM.

-- 10/15/2012 --
An issue began around 8:45AM that caused some users to receive a 404 error after entering their credentials. Our server team investigated and resolved the issue by 9:45AM.

-- 10/5/2012 -- 
Descriptive Analytics update was delayed but resolved.

-- 10/1/2012 --

A problem caused delays in overnight import processing. All files were queued and caught up by midday. 

-- 9/28/2012 -- 
Descriptive Analytics update was delayed but resolved.

-- 9/24/2012 --
An issue causing workflow to stop process was resolved at 7AM on Monday morning. All workflow processing had caught up by 2:00PM.

-- 9/20/2012 --

1:30 PM -- All files have now been processed and imports are running as expected.
11:05 AM -- Imports are processing as normal now. Any imports that have been queued up should process shortly. 
9:20 AM -- There are currently delays in processing import files. Files are queued and will be imported as received. Updates will be available later today.

-- 9/5/2012 --

4.45PM -- Workflow is back to operating at full speed, but may take an hour or two to get through any backlog.
2:20PM -- Workflow is processing slower than normal due to a problem with a server. We are working to get processing back up to full speed.

-- 8/21/2012 --

12:25PM -- A problem with the data center network caused the database server to initiate a fail over. This also rendered supporting websites and functions unavailable during the fail over event.
12:39PM -- Service was restored as the network issue was resolved. Reports remain unavailable due to this outage.
1:15PM -- A lingering problem on one firewall caused slow performance for some users. This is now resolved.
8/22 1:15AM -- Reports are now working again. 

-- 8/9/2012 --

1:15PM -- Workflow is all caught up and processing normally.
12:10PM -- Workflow is processing at normal speed again. We expect workflows to catch up and be back up to normal within a couple hours.
7:00AM -- Workflow is processing slowing than normal while we investigate the issue

-- 8/8/2012 --

4:15PM: The issue is now resolved. A certificate on one of our group of web servers failed which prevented users from logging in to Enrollment Manager. We have removed the web server from the group while it is repaired.
4:00PM: We are aware of an issuing affecting access for some users. The error appears after entering your user name and credentials on the log in page.

-- 8/6/2012 --

9:25 AM: Services have been restored for all organizations. We are still investigating the full cause of the issue and will post a report here with full details.
9:00 AM: We are investigating an issue which is causing some organization to not be able to log in. We will post updates here as we know more.

-- 7/27/2012 --

Previously reported issues with email marketing have been resolved.

-- 7/11/2012 --

11:30AM: The data center has confirmed that the issue has been resolved. We are awaiting further reports on the full cause of the issue.
10:30AM: Access has come back for some clients. We are working to get official resolution to the problem.
9:30AM: The issue has been confirmed as a problem with our data center's DNS servers and the ability to resolve the log in page. They are working to correct the problem.
8:00AM: Access to the Enrollment Manager log in page has been reported as sporadic for some clients. We are investigating the issue7/2 - An update by our email provider caused the "Send Bulk Email" button on an Outreach List to no longer work correctly. The button was updated to work properly with the new code and was deployed on 7/3.

-- 6/18/2012 --

7:56AM-9:15AM EST: Our Core Data Center experienced an internet outage affecting all services. Reports functionality were delayed for an additional time while the service was brought back up.

This issue was caused by a failed disk that caused the system to become hung. A firmware fix was applied on 6/24 to keep the issue from occurring again.


Comments