• Not Answered

Rebuilding the Reporting database

I am rebuilding my reporting database based on the steps outlined in the xDB Upgrade and Conversion Guide.  This is my first time to perform this action and it has been running for almost 3 hours.  This is not a production site, it's only a dev site, so there shouldn't be that much data...

How long does this rebuild process normally take?

Thanks!

Jason

7 Replies

  • I don't remember exactly time - but rebuild process takes long time
  • Hi Jason,

    Can I ask you a few questions to better understand the issue?

    - How much records do you have in XDB?
    - Do you see any errors in the logs?
    - What does the rebuildreportingdb.aspx page tell you? I shows the name of the current step that's running. Does it show "History processing"?
    - Have you tried increasing threads in aggregation configs?


    -alex
  • In reply to Alex Shyba:

    - looks like I only have 1265 records in the associated sitecoreInstance_analytics database.
    - The only error that appeared in my log file after I kicked off the process was (not sure if this is even related):
    ManagedPoolThread #19 2015:06:16 17:09:59 ERROR Cannot parse the time interval specified in the 'IntervalTokeep' option to the desired format 'hh:mm:ss' for the CleanupEventQueue agent. The value specified in the DaysToKeep option will be used instead – the specified value is '1' days. The specified value in the IntervalToKeep option is ''.
    - The rebuildreportingdb.aspx page stayed on HistoryProcessing for 18 hours before I just cancelled the job.
    - Which of the *.Aggregation*.config files should I modify to increase the number of threads?

    Thanks!
    Jason
  • In reply to Jason Wilkerson:

    Hi Jason,

    With that much data, I doubt you are lacking threads :-) I should finish relatively quickly. I bet the problem is somewhere else, something chokes up.

    It is rather strange that you don't see any other errors besides the one about CleanupEventQueue, that one is unrelated.

    Have you tried enabling <log4net /> appender to DEBUG? It will show a lot of more stuff going on during historic rebuild.

    Also, I would suggest is to identify the part of the platform that is critical for you to see the data. There are a lot of components that feed off the <interactions /> pipeline, and something could be blocking the process.

    For example, if it's Experience Analytics you are primarily interest in, you can comment out all other processors within <interactions /> pipeline.

    There could be a few. And those are scattered around various include configs. /sitecore/admin/showconfig.aspx will help finding those configs.

    This way you could at least isolate the issue.

    Finally, if the data you have in XDB is not sensitive, I would recommend filing the ticket and asking the Customer Service to reproduce and help troubleshooting.

    For the multi-threaded aggregation, the closes I've found is this article:

    Personally, I do the following (unofficial):

    1. Sitecore.Analytics.Processing.Aggregation.Services.config

    <aggregator type="Sitecore.Analytics.Core.BackgroundService">

      <param desc="agentName">aggregation/aggregator</param>

      <Interval>0.00:00:15</Interval>

      <MaxThreads>4</MaxThreads>

    </aggregator>

    <history type="Sitecore.Analytics.Core.BackgroundService">
      <param desc="agentName">aggregation/historyWorker</param>
      <Interval>0.00:00:15</Interval>
      <MaxThreads>4</MaxThreads>
    </history>

    2. Sitecore.Analytics.Processing.Services.config (not 100% sure if this one is needed)

    <taskagent type="Sitecore.Analytics.Core.BackgroundService">
      <param desc="agentName">processing/taskAgent</param>
      <Interval>0.00:00:15</Interval>
      <MaxThreads>4</MaxThreads>
    </taskagent>

    Please let me know if anything here helps at all.

  • In reply to Alex Shyba:

    My rebuilds are clocking in around 1 contact per second on a fairly powerful workstation using default settings with Sitecore and Mongo running locally. At that rate it could take months with millions of contacts. Increasing the aggregation threads speeds this up tremendously, but I noticed it can easily peg your CPU at 100% if you go too high. I can see why Sitecore went with a scale-out approach for the processing servers.
  • In reply to Hunziker Derek:

    8.0 Update-4 brings bulk aggregation. That should help getting experience analytics data in much faster. In the preliminary tests 15x boost in some cases.
  • Having exact same problem here. Can't find an answer. Was there a follow up to this? Support article?