What would cause a large difference between a "job submit time" and "job start time" in publishing?

[Dobi]

I spent a bit of time looking at detailed logs of publish jobs and comparing them to data that's available in Publishing Statistics from the WVS monitor. I can connect the dots to all the values in the statistics except for the "Job Submit time". I understand that's when the publishing was triggered (say through a check-in). What would cause for a large difference between it and "Job start"? Shouldn't that be pretty much instant? 

[avillanueva]

Hard to tell from that screen shot but jobs are submitted on check in correct. However, its a queue. If there are other jobs that are running, it will do those first before getting to the one shown. Was the queue empty at the time you checked in? Was there another job running?

[Dobi]

Yes, job is submitted on check in. The queue was mostly empty (I can look at another job to confirm). 

In this case I have a Job Submit time of 17:18. The job doesn't start (Job Start time) until 20:35 - some 3 hours later and that's when it gets added to the queue. Once it is in the queue (Worker Start time) it'll sit there until there's an available worker. For this job, there was a worker available not long after (this was 3rd in the queue so it sat there some ~3 minutes before the download started). 

I'm not sure why there is a 3 hour delay between job submit and job start. Wouldn't job submit send the job to an appropriate queue more or less right away? 

[HelesicPetr]

Hi @Dobi 

Does the worker have a setting to be available in period time to do publish job ?

That can explain the delay.

Or the queues were stopped and someone start them. 

Or Worker failed and were stopped and someone start it

PetrH

[Dobi]

@HelesicPetr 

The workers are on all the time (no settings to restrict availability). 

The queues were running. 

The workers could have failed (at least one of them is less than stable) but not all of them. There wasn't a point in time in which none of the workers were available. 

[Dobi]

Even looking at this right now:

There are no waiting entries in any of the queues and all the workers are available and online. Jobs going through now have no difference in time between Job Submit and Job Start. 

Earlier in the morning/late evening: There was a series of jobs that was submitted at 23:30(ish) but the jobs didn't start until around 00:30 so an hour later. There were a lot of jobs though so there was a delay in getting things published right away. I still don't understand why when a job is submitted it wouldn't immediately go to the queue and then just sit there. 

What would cause a delay in parts getting to the queue? 

[HelesicPetr]

Hi @Dobi 

What queue you are talking? A Process queue or schedule one?

There are schedule queues named PublishQueue L,M,H where the object is added to publish and wait until the process queue is available

Process queues are PublishQueue1-n where the publication is done. Process PublishQueue usually process just one object in same time. Only failed or finished jobs are there. depends how system is configured 

PetrH

[Dobi]

@HelesicPetr that's a good question and I'm not sure I have an answer. 

Publishing jobs break down to priority/schedules queues based on type/source definitions. So PublishQueue L, M, H. 

In my case these jobs go to the low priority queue. 

I have 4 publishers and 5 process queues. 

Is the order of operations then:

1. Job submit
   1. Determine type/source schedule queue (L, M, H) and put job there. There can be waiting entries in these queues.
   2. Job waits until it's ready to be moved to a process queue (1 through 5). Why doesn't it just go to a process queue right away? 
      1. What triggers a job to move from a schedule queue to a process queue? 
2. Job start
   1. Make temp workspace
   2. Determine authoring application
   3. Add job to temp workspace
   4. Process necessary conversion requirements (from publish rules)
   5. Send file to next available process queue
3. Worker start
   1. Job waits in process queue for available worker
4. Worker start file transfer
   1. Worker is assigned to the job
   2. Begin download etc. 

Is that about right for order of operations? 

If I want to reduce process queue wait time, I should have more workers or faster workers (so that jobs get through more quickly). 

If I want to reduce the time it takes for jobs to go from the schedule queue to the process queue, what do I do?