How to limit synapse spark to only send WARN/ERROR logs to log analytics workspace ?

@user_unknown

Thanks for the question and using MS Q&A platform.

To limit the logs sent to your Log Analytics workspace to only WARN and ERROR levels, you can adjust the log4j configuration in your Synapse Spark application. Here are the steps you can follow:

• Create a file named log4j.properties with the following content:

    log4j.rootCategory=WARN, console
    log4j.appender.console=org.apache.log4j.ConsoleAppender
    log4j.appender.console.target=System.err
    log4j.appender.console.layout=org.apache.log4j.PatternLayout
    log4j.appender.console.layout.ConversionPattern=%d{ISO8601} [%t] %-5p %c %x - %m%n
  

• You can upload this file to a location accessible by your Spark job, such as Azure Blob Storage or Azure Data Lake Storage.
• Add the following Spark configuration to your job to point to the custom log4j.properties file:

    spark.conf.set("spark.driver.extraJavaOptions", "-Dlog4j.configuration=file:/path/to/log4j.properties")
    spark.conf.set("spark.executor.extraJavaOptions", "-Dlog4j.configuration=file:/path/to/log4j.properties")
    
  

  
• Ensure that your Spark job is submitted with the above configurations. This will limit the logs sent to Log Analytics to only WARN and ERROR levels.

If you continue to experience issues, you might want to double-check the path to the log4j.properties file and ensure that it is correctly referenced in your Spark configuration

I hope the above steps will resolve the issue, please do let us know if issue persists. Thank you

@phemanth thank you for your quick response. I will try it out later today.

I am also reading online that spark3.3+ use log4j2. Does that change things for me? as I am using spark3.4

@phemanth I tried your suggested steps, but it didn't work. I still see info logs.

I have passed javaoptions like:
-Dlog4j.configuration=abfss://mystoragefs1@mystorage.dfs.core.windows.net/log4j.properties

@phemanth I tried it, but it didn't work. I still see info logs.

I passed abfss path like:
-Dlog4j.configuration=abfss://mystoragefs1@mystorage.dfs.core.windows.net/log4j.properties

@user_unknown

Thanks for information

Starting from Spark 3.3, the logging framework has been updated to use log4j2 instead of log4j1

This means you'll need to adjust your configuration accordingly.

Here's how you can set up log4j2 for Spark 3.4:

1. Create a log4j2.properties file with the following content:

      status = error
      name = PropertiesConfig
      filters = threshold
      filter.threshold.type = ThresholdFilter
      filter.threshold.level = warn
      appenders = console
      appender.console.type = Console
      appender.console.name = STDOUT
      appender.console.layout.type = PatternLayout
      appender.console.layout.pattern = %d{ISO8601} [%t] %-5p %c %x - %m%n
      rootLogger.level = warn
      rootLogger.appenderRefs = stdout
      rootLogger.appenderRef.stdout.ref = STDOUT
   

   Upload this file to a location accessible by your Spark job, such as Azure Blob Storage or Azure Data Lake Storage.
2. Add the following Spark configuration to your job to point to the custom log4j2.properties file:

      spark.conf.set("spark.driver.extraJavaOptions", "-Dlog4j.configurationFile=file:/path/to/log4j2.properties") spark.conf.set("spark.executor.extraJavaOptions", "-Dlog4j.configurationFile=file:/path/to/log4j2.properties")
   

   Ensure your Spark job is submitted with the above configurations. This will limit the logs sent to Log Analytics to only WARN and ERROR levels.

If you encounter any issues, double-check the path to the log4j2.properties file and ensure it is correctly referenced in your Spark configuration.

I hope the above steps will resolve the issue, please do let us know if issue persists. Thank you