New Security Analytics Service: Finding and Fixing Risk in Office 365

Article
08/03/2016

Microsoft is pleased to announce the preview availability of a new security analytics service called the Office 365 Secure Score. The Secure Score is a security analytics tool that will help you understand what you have done to reduce the risk to your data in Office 365, and show you what you can do to further reduce that risk. We think of it as a credit score for security. Our approach to this experience was very simple. First, we created a full inventory of all the security configurations and behaviors that our customers can do to mitigate risks to their data in Office 365 (there are about 77 total things that we identified). Then, we evaluated the extent to which each of those controls mitigated a specific set of risks and awarded the control some points. More points means a more effective control for that risk. Lastly, we measure the extent to which your service has adopted the recommended controls, add up your points, and present it as a single score.

The core idea is that it is useful to rationalize and contextualize all of your cloud security configuration and behavioral options into one simple, analytical framework, and to make it very easy for you to take incremental action to improve your score over time. Rather than constructing a model with findings slotted into critical, moderate, or low severity, we wanted to give you a non-reactive way to evaluate your risk and make incremental changes over time that add up to a very effective risk mitigation plan.

The Office 365 Secure Score is a preview experience, so you may find issues, and you will note that not all of the controls are being measured. Please share any issues on the Office Network Group for Security. You can access the Secure Score at https://securescore.office.com.

The Secure Score does not express an absolute measure of how likely you are to get breached. It expresses the extent to which you have adopted controls which can offset the risk of being breached. No service can guarantee that you will not be breached, and the Secure Score should not be interpreted as a guarantee in any way.

Your Secure Score Summary

The first, most important piece of the Secure Score experience is the Score Summary. This panel gives you your current Secure Score, and the total number of points that are available to you, given your subscription level, the date that your score was measured, as well as a simple pie chart of your score. The denominator of your score is not intended to be a goal number to achieve. The full set of controls includes several that are very aggressive and will potentially have an adverse impact on your users' productivity. Your goal should be to optimize your action to take every possible risk mitigating action while preserving your users' productivity.

Risk Assessment

While the Secure Score is framed as a 'gamification' of your security, it is important to recognize that every action you take will mitigate a real world threat. This panel shows you the top threats for your tenancy, given your particular configuration and behaviors. Make sure you read about and understand the risks you are mitigating every time you take an action.

Compare Your Score

The Office 365 Average Secure Score is calculated from every Office 365 customer's Secure Score. You can use this panel to get a better sense of how your score stacks up against the average. The specific controls that are passed by any given customer are not exposed in the average, and your Secure Score is private. Note that the Average Secure Score only includes the numerator of the score, not the denominator. So, the average points may be higher than you can achieve because there are points in controls associated with services that you have not purchased.

Take Action

Helping you figure out which actions to take to improve your score is the purpose of the Secure Score. There are three basic parts to the experience:

First, there is the modeler. Use the slider to figure out how many actions you want to review. Sliding to the left will reduce the number of actions in your list below, sliding to the right will increase the number. Each tick of the slider will add one control to the list. The target score shows you how much your score will increase if you take all the actions in the queue.

Second is the action pane. When you open this, you will see a description of the control, explaining why we think it is an effective mitigation, and what we observed about your configuration. We'll also show you some details about the control such as the category (account, device, data), what the user impact of the action is (low or moderate) as well as your measured score. Clicking Learn More will open a fly-out pane that will walk you through taking the desired action.

Thirdly, you will see a remediation pane fly-out that explains exactly what you are about to change, and how it will affect your users. Eventually, the Launch Now link (which takes you to a separate security center now) will allow you to make the desired change right from the Secure Score experience.

Score Analyzer

Since the Secure Score experience is restricted to users that have been designated a Global Tenant Administrator, we wanted to make it easy for admins to analyze and report to their executives and stakeholders their progress on risk mitigation over time. The Score Analyzer experience allows you to review a line graph of your score over time, to export the audit of your control measurements for the selected day to either a PDF or a CSV, and to review what controls you have earned points for, and which ones you could take action on.

What's Next

As mentioned, the Office 365 Secure Score is in a preview release. Over the coming months you will see us continue to add new controls, new measurements, and improvements to the remediation experiences. If you like what you see, please share with your network. If you see something we can improve, please share it with us on the Office Network Group for Security. We're looking forward to seeing your scores go up, and making the Secure Score experience as useful, simple, and easy as it can be.

Comments

Anonymous
August 16, 2016
Is it possible to assign permissions to non-global admins so they can login to https://securescore.office.com/ and have "view only" access? Our security team is interested in viewing this data on a regular basis.
Anonymous
August 16, 2016
BRILLIANT
Anonymous
August 17, 2016
Interesting tool. How often does it check the setting? Is there a way i can force it to recheck the settings? I enabled MFA for global admins yesterday and today it is still showing as an open control.
- Anonymous
  August 22, 2016
  Hey Vivek,Thanks for the feedback. The score is re-calculated once per day at 1am PST. There isn't currently a way to re-check the settings, but we're hoping to be able to do that in the future (or to decrease the latency). Did the MFA credit eventually come through?Thanks!Brandon
- Anonymous
  October 06, 2016
  This is looking really good.I didn't see an update the next day, so didn't go back for a few days. I'll check everyday now because the check did occur.Quite a few tasks are prefixed with [Not Scored], I think these are date based checks, where the action needs to be weekly, etc. I went to azure AD connect reports and ran them, come back next day and still nothing. will check back tomorrow.
  - Anonymous
    October 06, 2016
    Thanks for the follow-up, Mark. The [Not Scored] controls are items that we think you should do, but we haven't completed sourcing the data in the backend and calculating the score for that control. You can verify your setting, but we aren't able to award points until the score is...scored. :) Thanks!Brandon Koeller
Anonymous
August 18, 2016
This doesn't appear to be compatible with PIM-enabled accounts, even when they've been stepped up to Global Admin.
- Anonymous
  August 22, 2016
  Hey Tatham,Thanks for the feedback. This is likely a function of the calculation cycle the score currently operates on (which is to run once per day). If the PIM elevation is only for 4 hours, it will be relatively unlikely that it catches the account while elevated. We do our calculation at 1am PST daily. Thanks!Brandon
Anonymous
August 26, 2016
It's a little odd that the top suggestion is to enable MFA for Tenant Admins, when Microsoft doesn't have a working solution for connecting to Exchange Online via Powershell on a MFA-enabled admin account.
- Anonymous
  April 10, 2017
  Hey Matthew,I reached out offline with a response back in September, but thought I would follow-up here with the official method for accessing Exchange Online remote powershell with MFA enabled: https://technet.microsoft.com/en-us/library/mt775114(v=exchg.160).aspxThanks,Brandon Koeller
Anonymous
September 02, 2016
Would you need the E5 security workloads (ATM and ASM) to get a full score?
- Anonymous
  September 02, 2016
  The comment has been removed
Anonymous
October 04, 2016
Nice. We're finishing our Office 365 setup and this tool helps us not miss anything big (or small). Is this the best place to provide feedback? There are discrepancies between our Office 365 settings and what the tool reports, even considering the once-a-day recalculation. Thank you!
- Anonymous
  October 06, 2016
  Hi Paul,Thanks for the feedback. Glad the tool is working for you. Please either post feedback here, in the Office Network discussion board (https://techcommunity.microsoft.com/t5/Security-Privacy-Compliance/Announcement-Office-365-Secure-Score-Released-to-Public-Preview/m-p/5429#M13), or send email to o365securescore@microsoft.com. Would love to learn what the discrepancies are that you are seeing!Thanks,Brandon Koeller
  - Anonymous
    October 10, 2016
    Hi Brandon. Thank you for the reply and I will also post my findings at the site you mentioned. Hopefully I'm overlooking something.1. From the tool: "We found that you had 18 mailboxes of 25 with audited enabled." When I run get-mailbox with and without -SoftDeletedMailbox a total of 24 mailboxes are returned. Of those 24, auditing was enabled on 19 of them.2. From the tool: "You should set your Exchange Online Spam Policies to copy and notify someone when a sender in your tenant has been blocked for sending excessive or spam emails. ... We found that your outbound spam policy is configured to False." It's my understanding that the outbound spam policy cannot be disabled, and in our current policy I have both "copy suspicious messages" and "Send a notification to the following email address or addresses when a sender is blocked for sending outbound spam." enabled and populated with smtp addresses.Thank you for taking feedback. :)
Anonymous
October 05, 2016
Excellent tool to verify the security level of our configuration in Office 365!I understand from our tenant admins that I have to be an admin to access the score.As security specialist I would like to have display access to the score and recommendations to follow the progress of our efforts to secure Office 365 – is that possible?/Peter
- Anonymous
  October 06, 2016
  Hi Peter,Thanks so much for the feedback. Is there an Azure Active Directory role that you think would be appropriate to give access to the Secure Score? We agree that we want it to be available to more roles, but there isn't an AAD role for Security Admins (yet). If we also gave Exchange and Sharepoint admins access would that work?Thanks!Brandon Koeller
  - Anonymous
    October 12, 2016
    There is a "Security Administrator" role within the Security & Compliance Centre (https://protection.office.com/#/permissions) which seems like it would be appropriate to grant access to this feature. Speaking of the Centre, are there plans to bring this feature under that banner?
    - Anonymous
      November 08, 2016
      Hey Scott. Thanks for reaching out! We do plan to leverage the Security Administrator role and do expect to be able to link to the Secure Score experience from the Security and Compliance Center. Thanks! Brandon Koeller
  - Anonymous
    October 13, 2016
    How about the Security Reader and Security Administrator roles? I think there is significant overlap in what these roles can do and how Secure Score is likely to be used.
  - Anonymous
    October 19, 2016
    I am going to chime in and say no that will not work. There needs to be a security specific role for this without giving our security folks the keys to our cloud kingdom. Global Admin privileges fall way outside of their function in most organizations, including ours.
Anonymous
October 12, 2016
Really cool stuff! It doesn't seem to care about policies configured in Intune though..., any plans to cover this?
- Anonymous
  November 08, 2016
  The comment has been removed
Anonymous
November 01, 2016
Can you run this as a reseller as a Global Admin on the account? Or do you have to be a Tenant Admin?
- Anonymous
  November 01, 2016
  I should have said, as a Delegated Admin... sorry.
- Anonymous
  November 08, 2016
  Has to be a tenant admin for now! More roles coming soon though. Thanks, Brandon Koeller
Anonymous
November 01, 2016
I should have said, as a Delegated Admin... sorry.
Anonymous
March 12, 2017
There should be a way to select the criteria that's important to your company and then try to reach that goal. Let's face it the score is meaningless if it's a comparison to everything, and all companies have different policies. It's neat but and a good place for some recommendations but not entirely useful from a ranking point of view.
- Anonymous
  April 10, 2017
  Hey Bryan,Thanks for reaching out, and sorry for the trouble. The mobile device management controls are in a bit of a weird state. We have instrumentation for the native O365 MDM controls, but not quite yet for the InTune product. So, you get points for having adopted O365 MDM, but not for adopting InTune, but the controls actually point you to adopting InTune. We're working with the InTune team now to get access to the telemetry and correctly score adoption of InTune. Stay tuned!Thanks,Brandon Koeller
Anonymous
March 16, 2017
Hi Brandon, we have problem with "enable mobile device management services". Intune is enabled since at least 1 year and we don't have any point for it.. When we click on the link "launch now" of the action we received the following message (translate from french): you don't need to configure mobile device management for office 365, your organization is already protected by Microsoft intune".The score method seems to not find Intune, do you have an idea? I can send you tenant information if needed..
Anonymous
April 06, 2017
Some Recommended actions were completed. Other actions listed as incomplete are actually enabled. Secure Score does not recognize some actions already complete as completed with an associated score. Other completed actions taken using the tool do no increase Secure Score. This is an important tool for my organization to use but it is not working as advertised.
- Anonymous
  April 10, 2017
  Hey Nashtek,Thanks for reaching out. Any action that is currently labeled as [Not Scored] will, at least for the moment, not reflect an increase in your score if you do the action. The action itself is possible, and your actual security will improve, but we haven't been able to instrument the telemetry in the backend quite yet. Sorry for the trouble! Very glad to hear that the tool is important for your organization. We're working hard to make sure all the controls get instrumented.Thanks,Brandon Koeller

Share via