Run-Time HW/SW Scheduling of Data Flow Applications on Reconfigurable Architectures
© Fakhreddine Ghaffari et al. 2009
Received: 1 March 2009
Accepted: 7 October 2009
Published: 23 November 2009
Skip to main content
© Fakhreddine Ghaffari et al. 2009
Received: 1 March 2009
Accepted: 7 October 2009
Published: 23 November 2009
This paper presents an efficient dynamic and run-time Hardware/Software scheduling approach. This scheduling heuristic consists in mapping online the different tasks of a highly dynamic application in such a way that the total execution time is minimized. We consider soft real-time data flow graph oriented applications for which the execution time is function of the input data nature. The target architecture is composed of two processors connected to a dynamically reconfigurable hardware accelerator. Our approach takes advantage of the reconfiguration property of the considered architecture to adapt the treatment to the system dynamics. We compare our heuristic with another similar approach. We present the results of our scheduling method on several image processing applications. Our experiments include simulation and synthesis results on a Virtex V-based platform. These results show a better performance against existing methods.
One of the main steps of the HW/SW codesign of a mixed electronic system (Software and Hardware) is the scheduling of the application tasks on the processing elements (PEs) of the platform. The scheduling of an application formed by N tasks on M target processing units consists in finding the realizable partitioning in which the N tasks are launched onto their corresponding M units and an ordering on each PE for which the total execution time of the application meets the real-time constraints. This problem of multiprocessor scheduling is known to be NP-hard [1, 2], that is, why we propose a heuristic approach.
Many applications, in particular in image processing (e.g., an intelligent embedded camera), have dependent data execution times according to the nature of the input to be processed. In this kind of application, the implementation is often stressed by real-time constraints, which demand adaptive computation capabilities. In this case, according to the nature of the input data, the system must adapt its behaviour to the dynamics of the evolution of the data and continue to meet the variable needs of required calculation (in quantity and/or in type). Examples of applications where the processing needs changes in quantity (the computation load is variable) come from the intelligent image processing where the duration of the treatments can depend on the number of the objects in the image (motion detection, tracking, etc.) or of the number of interest areas (contours detection, labelling, etc.).
We can quote also the use of run time of different filters according to the texture of the processed image (here it is the type of processing which is variable). Another example of the dynamic applications is video encoding where the run-length encoding (RLE) of frames depends on the information within frames.
For these dynamic applications, many implementation ways are possible. In this paper we consider an intelligent embedded camera for which we propose a new design approach compared to classical worst case implementations.
Firstly, to evaluate the partitioning result. After each modification of the tasks implementations we need to know the new total execution time. And this is only possible by rescheduling all the tasks.
Secondly, by modifying the scheduling result we can obtain a better total execution time which meets the real-time constraint without modifying the partitioning. This is because the characteristics of the tasks (mainly execution time) are modified according to the nature of the input data.
In that context, the choice of the implementation of the scheduler is of major importance and depends on the heuristic complexity. Indeed, with our method the decisions taken online by our scheduler can be very time consuming. A software implementation of the proposed scheduling strategies will then delay the application tasks. For this reason, we propose in this work a hardware implementation for our scheduling heuristic.
With this implementation, the scheduler takes only few clock cycles. So we can easily call the scheduler at run time without penalty on the total execution time of the application.
The primary contribution of our work is the concept of an efficient online scheduling heuristic for heterogeneous multiprocessor platforms. This heuristic provides good results for both hardware tasks (onto the FPGA) and software tasks (onto the targeted General Purpose Processors) as well as an extensive speedup through the hardware implementation of this scheduling heuristic. Finally, the implementation of our scheduler allows the system to adapt itself to the application context in real time. We have simulated and synthesized our scheduler by targeting a FPGA (Xilinx Virtex 5) platform. We have tested the scheduling technique on several image processing applications implemented onto a heterogeneous target architecture composed of two processors coupled with a configurable logic unit (FPGA).
The remainder of this paper is organized as follows. Section 2 presents related works on hardware/software scheduling approaches. Section 3 introduces the framework of our scheduling problem. Section 4 presents the proposed approach. Section 5 shows the experimental results, and finally Section 6 concludes this paper.
The field of study which tries to find an execution order for a set of tasks that meets system design objectives (e.g., minimize the total application execution time) has been widely covered in the literature. In [4–6] the problem of HW/SW scheduling for system-on-chip platforms with dynamically reconfigurable logic architecture is exhaustively studied. Moreover several works deal with scheduling algorithm implemented in hardware [7–9]. Scheduling in such systems is based on priorities. Therefore, an obvious solution is to implement priorities queues. Many hardware architectures for the queues have been proposed: binary tree comparators, FIFO queues plus a priority encoder, and a systolic array priority queue . Nevertheless, all these approaches are based on a fixed priority static scheduling technique. Moreover most of the hardware proposed approaches addresses the implementation of only one scheduling algorithm (e.g., Earliest Deadline First) [9, 10]. Hence they are inefficient and not appropriate for systems where the required scheduling behavior changes during run time. Also, system performance for tasks with data dependent execution times should be improved by using dynamic schedulers instead of static (at compile time) scheduling techniques [11, 12].
In our work, we propose a new hardware implemented approach which computes at run-time tasks priorities based on the characteristics of each task (execution time, graph dependencies, etc.). Our approach is dynamic in the sense that the execution order is decided at run time and supports a heterogeneous (HW/SW) multiprocessor architecture.
The idea of dynamic partitioning/scheduling is based on the dynamic reconfiguration of the target architecture. Increasingly FPGA [13, 14] offers very attractive reconfiguration capabilities: partial or total, static or dynamic.
The reconfiguration latency of dynamically reconfigurable devices represents a major problem that must not be neglected. Several references can be found addressing temporal partitioning for reconfiguration latency minimization . Moreover, configuration prefetching techniques are used to minimize reconfiguration overhead. A similar technique to lighten this overhead is developed in  and is integrated into an existing design environment. A prefetch and replacement unit modifies the schedule and significantly reduces the latency even for highly dynamic tasks.
In fact, there are two different approaches in the literature: the first approach reduces reconfiguration overhead by modifying scheduling results.
The second one distinguishes between scheduling and reconfiguration. The reconfiguration occurs only if the HW/SW partitioning step needs it. The scheduling algorithm is needed only to validate this partitioning result.
After partitioning, the implementation of each task is unchanged and configuration is not longer necessary. Scheduling aims at finding the best execution time for a given implementation strategy. Since scheduling does not change partitioning decision it does not take reconfiguration time into account.
In this paper, we focus only on the scheduling strategy in the second case. We assume that the reconfiguration aspects are taken into account during the HW/SW partitioning step (decision of task implementation). Furthermore we addressed this last step in our previous works .
Each hardware task (implemented on the RCU) occupies a tile on the reconfigurable area . The size of the tile is the same for all the tasks to facilitate the placement and routing of the RCU. We choose, for example, the tile size of the task which uses the maximum of resources on the RCU (we designate by "resource" here the Logic Element used by the RCU to map any task).
The RCU unit can be reconfigured partially or totally. Each hardware task is represented by a partial bitstream. All bitstreams are memorized in the contexts memory (the shared memory between the processors and the RCU in Figure 1). These bitstreams will be loaded in the RCU before scheduling to reconfigure the FPGA according to run-time partitioning results . The HW/SW partitioning result can change at run time according to temporal characteristics of tasks . In  we proposed an HW/SW partitioning approach based on HW SW and SW HW tasks migrations. The theory of tasks migrations consists in accelerating the task(s) which become critical by modifying their implementations from software units to hardware units and to decelerate the tasks which become noncritical by returning them to the software units.
After each new HW/SW partitioning result, the scheduler must provide an evaluation for this solution by providing the corresponding total execution time. Thus it presents a real-time constraint since it will be launched at run time. With this approach of dynamic partitioning/scheduling the target architecture will be very flexible. It can self-adapt even with very dynamic applications.
Each node of the DFG represents a specific task in the application. For each task there can be up to three different implementations: Hardware implementations (HW) placed in the FPGA, Software implementations running on the master processor (MS), and another Software implementation running on the slave processor (SL).
Each node of Figure 2 is annotated with two data: one about the implementation (MS or SL or HW) and the other is the execution time of the task. Similarly each edge is annotated with the communication time between two nodes (two tasks).
Texe (execution time),
Impl (implementation on the RCU or on the master processor or on the slave processor),
Nbpred (number of predecessor tasks),
The Nbsucc (number of successor tasks).
All the tasks of a DFG are thus modeled identically, and the only real-time constraint is on the total execution time. At each scheduler invocation, this total execution time corresponds to the longest path in the mapped task graph. It then depends both on the application partitioning and on the chosen order of execution on processors.
The applications are periodic. In one period, all the tasks of the DFG must be executed. In the image processing, for instance, the period is the execution time needed to process one image. The scheduling must occur online at the end of the execution of all the tasks, and when a violation of real-time constraints is predicted. Hence the result of partitioning/scheduling will be applied on the next period (next image, for image processing applications).
Our run-time scheduling policy is dynamic since the execution order of application tasks is decided at run time. For the tasks implemented on the RCU, we assume that the hardware resources are sufficient to execute in parallel all hardware tasks chosen by the partitioning step. Therefore the only condition for launching their execution is the satisfaction of all data dependencies. That is to say, a task may begin execution only after all its incoming edges have been executed.
For the tasks implemented on the software processors, the conditions for launching are the following.
The discharge of the software unit.
The task is in the waiting state when it waits the end of execution of one or several predecessor tasks. When a software processing unit has finished the execution of a task, new tasks may become ready for execution if all their dependencies have been completed of course.
The task can be stopped in the case of preemption or after finishing its execution.
The states of the processing units (SW, SL, and HW) in our target architecture are: execution state, reconfiguration state or idle state.
In the following, we will explain the principle of our approach as well as a hardware implementation of the proposed HW/SW scheduler.
As explained in Algorithm 1, the basic idea of our heuristic of scheduling is to take decision of tasks priorities according to three criteria.
Algorithm 1: Principle of our scheduling policy.
For all Software tasks do
Task with minimum ASAP will be chosen
If (Equality of ASAP)
Task with maximum urgency will be chosen
If (Equality of Urgency)
Compare Execution time
Task with maximum execution time will be chosen
The first criterion is the As Soon As Possible (ASAP) time. The task which has the shortest ASAP date will be launched first.
The second criterion is the urgency time: the task which has the maximum of urgency will have priority to be launched before the others. This new criterion is based on the nature of the successors of the task. The urgency criterion is employed only if there is equality of the first criterion for at least two tasks. If there is still equality of this second criterion we compare the last criterion which is execution time of the tasks. We choose the task which has the upper execution time to launch first.
We use these criteria to choose between two or several software tasks (on the Master or on the Slave) for running.
The urgency criterion is based on the implementation of tasks and the implementations of their successors. A task is considered as urgent when it is implemented on the software unit (Master or slave) and has one or more successor tasks implemented on other different units (hardware unit or software unit).
When a task has several successors with different implementations, the urgency is the maximum of execution times of the successors.
In general case, when the direct successor of task A has the same implementation as A and has a successor with a different implementation, then this last feedbacks the urgency to task A.
We show the scheduling result for case (a) when we respect the urgency criterion in Figure 3(d) and otherwise in Figure 3(e).We can notice for all the examples of DFG in Figure 3 that the urgency criterion makes a best choice to obtain a minimum total execution time. The third criterion (the execution time) is an arbitrary choice and has very rarely impact on the total execution time.
We can notice also that our scheduler supports the dynamic creation and deletion of tasks. These online services are only possible when keeping a fixed structure of the DFG along the execution. In that case the dependencies between tasks are known a priori. Dynamic deletion is then possible by assigning a null execution time to the tasks which are not active. and dynamic creation by assigning their execution time when they become active.
This scheduling strategy needs an online computation of several criterions for all software tasks in the DFG.
We can conclude that, in an embedded context, a software implementation of this strategy is thus incompatible with real-time constraints.
We describe in the following an optimized hardware implementation of our scheduler.
The DFG_IP_Sched (the middle part surrounded by a dashed line in the figure).
The DFG_Update (DFG_Up in the figure).
The MS_Manager (SWTM).
The Slave_Manager (SLTM).
The basic idea of this hardware architecture is to parallelize at the maximum the scheduling of processing tasks. So, at the most (and in the best case), we can schedule all the tasks of the DFG in parallel for infinite resources architecture.
We associate to the application DFG a modified graph with the same structure composed of the IP nodes (each IP represents a task). Therefore in the best case, where tasks are independent, we could schedule all the tasks in the DFG in only one clock cycle.
To parallelize also the management of the software execution times, we associate for each software unit a hardware module:
These two modules manage the order of the tasks executions and compute the processor execution time for each one.
A pointer in memory to the implementations of all the tasks. We have three kinds of implementation (RCU, Master, and Slave). With the signals SW and HW we can code these three possibilities.
The measured execution time of each task (Texe).
The Clock signal and the Reset.
The total execution time after scheduling all tasks (Texe_Total).
The signal All_Done which indicates the end of the scheduling.
Scheduled_DFG is a pointer to the scheduling result matrix to be sent to the operating system (or any simple executive).
The Nb_Task and the Nb_Task_Slave are the number of tasks scheduled on the Master and the number of tasks scheduled on the Slave, respectively. As noted here, these two signals were added solely for the purpose of simulation in ModelSim (to check the scheduling result). In the real case we do not need these two output signals since this information comes from the partitioning block.
The last one is the DFG_Up. This allows updating the results matrix after each scheduling of a task.
In the following paragraphs, we will detail each part of this architecture.
In this block there are components ( is the number of tasks in the application). For each task we associate an IP component which computes the intrinsic characteristics of this task (urgency, ASAP, Ready state, etc.). It also computes the total execution time for the entire graph.
For each task the implementation PE and the execution time are fixed, so the role of this IP is to calculate the start time of the task and to define its state. This is done by taking into account the state of the corresponding target (master, slave, or RCU). It then iterates along the DFG structure to determine a total execution ordering and to affect the start time.
This IP calculate also the urgency criterion of critical tasks according to the implementation and the execution time of their successors.
If the task is implemented on the RCU it will be launched as soon as all its predecessors will be done. So the scheduling time of hardware tasks depends on the number of tasks that we can run in parallel. For example, the IP can schedule all hardware tasks that can run in parallel in a one clock cycle.
For the software tasks (on the master or on the slave) the scheduling will take one clock cycle per task. Thus the computing time of the hardware scheduler only depends on the result of the HW/SW partitioning.
The Signal MIN_ASAP_TASKS represents all the tasks "Ready" and having the same minimum values of time ASAP.
The signal MAX_CT_TASKS represents all the tasks "Ready" and having the same maximum of urgency. The tasks which have the two preceding criteria will be represented by the Tasks_Ready signal. The Task_Scheduled signal determines the only software task which will be scheduled. With this signal, it is possible to choose the good value of signal TEXE_SW and to give the new value of the SW_Total_Time signal thereafter. A single clock cycle is necessary to schedule a single software task.
By analogy the Slave_Manager block has the same role as the SW_Manager block. From scheduling point of view there is no difference between the two processors.
The OS browses the matrix row by row. Whenever it finds a "1" it passes the task whose number corresponds to the column in the waiting state. At the end of a task execution the corresponding waiting tasks on each units will become either Ready or Running.
A task will be in the Ready state only when all its dependencies are done and that the target unit is busy. Thus there is no Ready state for the hardware tasks.
It should be noted that if the OS runs on the Master processor, for example, this later will be interrupted each time to execute the OS.
a motion detection application,
an artificial extension of this detection application,
a robotic vision application.
The second case study is a complex DFG which contains different classical structures (Fork, join, sequential). This DFG is depicted in Figure 12. It contains twenty tasks. Each task can be implemented on the Software computation unit (Master or Slave processor) or on the Reconfigurable RCU. The original DFG is the model of an image processing application: motion detection on a fixed image background. This application is composed of 10 sequential tasks (from ID 1 to ID 10 in Figure 12). We added 10 others virtual tasks to obtain a complex DFG containing the different possible parallel structures. This type of parallel program paradigm (Fork, join, etc.) arises in many application areas.
In order to test the presented scheduling approach, we have performed a large number of experiments where several scenarios of HW/SW partitioning results were analyzed.
As an example, Figure 12 presents the scheduling result when tasks 3, 4, 7, 8, 11, 12 17, 18, and 20 are implemented in hardware. As explained in Section 4.1 new dependencies (dotted lines) are added in the original graph to impose a total order on each processor. In this figure all the execution times are in milliseconds (ms).
We also leaded our experiments on a more dynamic application from robotic vision domain . It consists in a subset of a cognitive system allowing a robot equipped with a CCD-camera to navigate and to perceive objects. The global architecture in which the visual system is integrated is biologically inspired and based on the interactions between the processing of the visual flow and the robot movements. In order to learn its environment the system identifies keypoints in the landscape.
Throughout our experiments, we compared the result of our scheduler with the one given by the HCP algorithm (Heterogeneous Critical Path) developed by Bjorn-Jorgensen and Madsen . This algorithm represents an approach of scheduling on a heterogeneous multiprocessor architecture. It starts with the calculation of priorities for each task associated with a processor. A task is chosen depending on the length of its critical path (CPL). The task which has the largest minimum CPL will have the highest priority. We compared with this method because it is shown better than several other approaches (MD, MCP, PC, etc.) .
By comparing the first and second rows, our scheduling method provides consistent results.
The quality of the scheduling solutions found by our method and the HCP method is similar. Moreover, our method obtains better results for the complex Icam application. The HCP method returns an average total execution time equal to 69 milliseconds whereas our method returns only 58 milliseconds for the same DFG. For the icam_simple application, the DFG is completely sequential, so whatever the scheduling method the result is always the same. For the robotic vision application, we find the same total execution time with the two methods because of the existence of a critical path in the DFG which always sets the overall execution time. We also measured the execution overhead of the proposed scheduling algorithm when it is implemented in software (third row of Figure 14) and in hardware (forth row).
From Figure 15, it may be concluded that when the result of partitioning changes at runtime, then the computation time needed for our scheduler to schedule all the DFG tasks is widely dependent on this modification of tasks implementations. So:
the longest sequential sequence of tasks corresponds to the case where all tasks are on the Software (each task takes one clock cycle). This case corresponds to the maximum of schedule computation time,
the minimum schedule computation time depends on the DFG structure. The longest sequential sequence of tasks when all tasks are on the hardware (each task takes one clock cycle).
In our case (Figure 12) this sequence is formed by 10 tasks, so the minimum schedule computation time is equal to 10 clock cycles for this application.
The results confirm that a software implementation of our scheduling algorithm is incompatible with online scheduling. Instead, the hardware implementation proposed in the paper brings determinism, a better scalability, and an 20000 speedup.
Device utilization summary after synthesis.
Used logic utilization
Number of slices registers
Number of slice LUTs
Number of fully used Bit Slices
Number of bonded IOBs
Number of BUFG/BUFGCTRLs
The accuracy of the execution time values is defined by the size of the bus which must convey the information from module to another. The size of this bus is a very determinant parameter in the scheduler synthesis.
In the general case, the designer has to make a tradeoff between accuracy of performance measures (in this case the execution time) and the cost in number of hardware resources and the maximum frequency of the circuit.
Through the various results of synthesis, we confirm the effectiveness of the hardware architecture for our proposed scheduling method. With these three applications, we have swept most of existing DFGs structures: the sequential in the application icam_simple, the fork and join in the application icam_complex, and the parallelism in the application of robotic vision.
This scheduling heuristic gives better results than the HCP method. Moreover the proposed hardware architecture is very efficient in terms of resources utilization and scheduling latency.
These features allow our scheduling IP to run online and meet the needs of the dynamic nature of most today applications.
In this paper, we presented a complete run-time hardware-software scheduling approach. Results of our experiments show the efficiency of the adaptation of the scheduling to a dynamic change of the partitioning that can be due to a new mode of a dynamic application or to fault detection. As developed in this paper, a dynamic HW/SW scheduling approach has many advantages over static traditional approaches. In addition, the efficiency of our hardware implementation gives to our scheduler a minimal overhead in on line execution context.
Behaviour of the scheduler in dynamic situations.
Total execution time of application
Size of the scheduler IP
Need to resynthesize
Variation of execution time of tasks
Variation of partitioning results
Variation of the DFG structure (fork, join, etc.)
Variation of the application
Variation of the execution time precision
We show through this table in which case it is necessary to restrict the IP scheduler and resynthesize it and in which case the IP can adapt to the dynamic system.
Our future works consist in integrating our scheduling approach among the services of an RTOS for dynamically reconfigurable systems.
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.