Engineering Management System - Design Review Process

[oz_olly]

I'm interested to know how many teams operate an Engineering Management System (EMS) for the purposes of managing design review, vehicle configuration and the technical integrity of the car. What I am proposing to do within our team is establish an EMS which outlines a design review process that is initiated with a judgement of significance (JOS). The JOS will be based on technical risk (what are the consequences of the design failing or partially performing), cost and schedule risk. If a design is deemed significant then it would have to be reviewed by the student Technical Director (Chief Engineer) and approved by our technical Faculty Adviser, non-significant designs could be review by another design engineer and approved by the Technical Director.

The idea of the JoS has come from the Australian Defence Force Technical Airworthiness Management Manual which says:

All designs are required to be subject to a Judgement of Significance (JoS). This judgement is an assessment
of the technical risk introduced by the implementation of the design and must consider both the probability and
consequence of partial performance or failure of a design. Subsequent to a JoS, all designs shall be classed as either
Significant or Non-significant. Designs judged as Significant are subject to a more rigorous Design Review and
Approval process.

To an extent we have been running a vehicle maintenance log for some time but it has seen sporadic use. The idea here is to enter and unserviceability when something is broken, routine maintenance falls due etc and a corrective action has to be entered against the unserviceability prior to it being signed off as complete. The car would then not be allowed to operate unless all unserviceabilities were close out. We could then also have a Carried Forward Unserviceability where the Technical Director makes a risk assessment about operating the US unresolved.

The other key aspect of the EMS would be the configuration management of the car and associated assembly drawings with torque values and any particular assembly instructions forming a maintenance manual. As we make changes to the maintenance it would be updated.

I think there are many benefits of using a system like this but it has the danger of becoming administratively cumbersome, it should however help to overcome the reliance on having key team members present to do certain maintenance functions and the subsequent loss of knowledge when they graduate.

Thoughts and similar experiences would be appreciated.

Cheers

[thewoundedsoldier]

I think that a program like that really hinges on the strength of your team management and the structure you have in place, top-down.

Last year I was team manager (formula hybrid) and really learned what over-management means. I implemented a lot of protocol (some successful, some not so much). At the end of it all I've learned that management must be tailored to the team, not vice versa. Our school and student population generally have very little resources, so this year we are favoring a much smaller, tighter-knit team that focuses on design efficiency and communication. This is instead of spending time recruiting a massive team and segregating into several sub-teams with processes such as the one you mention. It was a systematic decision to avoid excess weight and help us use our resources as richly as possible. I hope this makes sense.

Overall it is totally school dependent. I look at teams like GFR and just gawk. How do they maintain that structure? I am sure that if you are even thinking about an approach like the one you mention, your school has a solid structure with a large, motivated team.


About the JoS in particular, it puts a smile on my face. I am a U.S. Marine and am all too familiar with having an acronym for nearly everything. The reason it is done so much in military and other government organizations is because of bureaucracy: people need to implement systems to continue justifying their paychecks. So they create these large, sprawling systems which may or may not be efficient when put into a formula student environment.

No matter how you slice it, I wish you luck and would like to hear what results you find at the end of the coming season.

[oz_olly]

In your case, opting for the more informal approach based on good communication between a few key team members, what have you done to help prevent entire load cases being missed during component analysis, incorrect FEA constraints etc. Do you look at or check each others work at all? Or is it more a case of trying to do your best and staying fairly conservative? The latter is the way our team has operaed more or less since the beggining but fortunately our technical faculty adviser has extensive industry experience ranging from aerospace to heavy rail and I usually run all designs that I would loosely term 'significant' past him for a bit of a reality check.

I do agree with you that alot of the military technical integrity frameworks are so rigorous that it guarantees the employment of thousands of public servants well into the future but I do think some of the concepts, applied in the most relaxed way possible could really help. Our team is definitely not Rennteam or GFR but I think implementing something like the above with minimal management impost would actually assist with knowledge transfer and better equip incoming Technical Directors, this is based on the fact we have a very flat team structure and currently delegate jobs based on ability rather than year level.

Cheers

[thewoundedsoldier]

I believe the best way to not miss errors is to have a small, tight-knit team that keeps each other in check. A large team requires the system you mentioned above and even with such a system in place, cracks exist. I have found that by not delegating specific sub-teams or sub-tasks, we all get to keep an equal eye on each other's work because we all have equal investment/ownership in every component. The moment you name John Doe the Suspension Lead, you rob Mary, Donald, and Alvin of any ownership in suspension design. This means they will be less inclined to inspect suspension carefully, just like that John Doe won't carefully inspect driveline or ergonomic designs.

I would like to say that a small, efficient model could compete with a large, bureaucratic model, but it doesn't appear to be true. All of the best teams look large on paper and website, and all of them have similar systems to what you propose. Our way of keeping it small is simply doing the best with what we have.

I would be interested to hear what other teams think, especially the SJSU FSAE team.

To actually answer your questions (I am the king of tangents...):

We do most loading analysis and optimization problems at the senior design level. These are presented in the senior design class so we try to tighten them up as much as possible. A lot gets missed. We have not yet established technical support from our faculty advisors, so there isn't much help there.

Your point about knowledge transfer is spot-on. Our way of incorporating it this year is to keep a binder which has part drawings and specs for every single component on the car, from engine to rodend. Like you have said indirectly, this leads to better knowledge amongst current members and eases the knowledge-transfer process. I would have guessed that a lot of schools do this, but in talking to most I find it is pretty rare...

When you say you have a flat team structure, do you mean that you are structured more horizontal than vertical? Last year I went to great lengths to ensure that we were well-structured vertically and it turned out to be a disaster. As a Recon Marine with strong appreciation for the chain of command, It took a lot of mistakes for me to learn that formula student is a project of peers. dammit i am going off on a tangent again...

[exFSAE]

JoS sounds like a DFMEA.

If I were to do FSAE again, the management and design review process would be the #1 thing I would target for change.

Our "review" was a CDR (critical design review) at the end of the semester with some faculty and alumni. Usually wound up being 1.5-2.0 hours. Unfortunately this amounts to a super brief overview of each system to fit the whole car into that time frame. To do it right, each system (suspension, powertrain, ergo, controls) should really have been about 2 hours worth. Furthermore, by that point in the year, you're too far down the critical path to really make any big design changes.

We had a "technical lead" during the year, but given that they also had to design parts on the car, there wasn't really sufficient time to overview every major system.

If I were to do it again, I'd break the car development down into a handful of teams. Vehicle dynamics, powertrain, electronics, chassis. Individual students may be on one or more teams.

I'd then have about an hour long weekly review scheduled with each subteam. Critical thing here would be to include the 'design lead' for the car, a faculty member, and ideally some alumni with industry experience. Go over timelines, budget allocation, design challenges, etc.

In this way, you're ensuring:

(a) There's some cross-functional integration, as the 'design lead' will see these reviews for everything on the car

(b) People are held accountable and are graded fairly, with the faculty member present.

(c) Good design decisions are being made and poor choices are identified early enough that they can be corrected.

[RobbyObby]

That is much the way our team is running this year and have run in the past. IMPO I think you need to have "subteams" to an extent. This way each part and system is designed with a greater efficiency. Those with specific talents towards one area of the overall design can put those to use better. In much the same way that I cant even begin designing an intake system based on choke flow, runner length, pulse tuning, and all that jazz, I dont expect our engine lead to know how to figure out the TR goal for our chassis based on suspension behavior and expectations. Even with a more horizontal team organization, I think you will eventually end up "semi-segregated", if thats the right word, since a team member is not gonna put in as much effort into a design that doesn't interest him, as opposed to designing a system that he cares about and has more knowledge with.

I am also a major proponent of the weekly design review system. You essentially kill two birds with one stone. As exFSAE mentioned, the poor design paths and choices are weeded out early. It also forces whoever designed the system to learn how to defend his design in front of a panel of "judges", which is exactly what they will have to do at Design Judging. Just sounding confident about your design and decisions shows alot to the judges.

[thewoundedsoldier]

Robby,

Would you agree that the segregation of tasks and analysis led to the endurance failure at comp last year?

An issue remained unresolved because of inefficiency in the system. A crack existed and murphy got ya. I only bring up this particular failure because it was a perfect example of subteams not communicating or working well enough together. If that problem was not left to just one guy, maybe other's would have noticed?

I would agree that design review is unquestionably a staple of a successful team. Last year our hybrid team did it monthly, while the FSAE team did it weekly. I kind of robbed the thread and made it about large teams vs. small teams and inefficiency vs. efficiency. Sorry.

[oz_olly]

I probably didn't explain well enough but my rationale for implementing the JoS is to use it as a decision point. If the design is significant the design must be accepted by the Technical Faculty Adviser, if it is non-significant then the student Technical Director can accept the design. Because the majority of designers on the team will be inexperienced the critical subsystems may have a predetermined JoS. Our design process would go something like this:

The conceptual design is developed between senior team members, perhaps some alumni and the faculty advisers. Hopefully our conceptual design would be enduring for several years at a time acting as strategic guidance for the design of the car. I would like to see a three year rolling conceptual design to ensure the team remains focussed and doesn’t reinvent the wheel every year.

Once the conceptual design is firmed up all the design briefs are developed and farmed out. The design briefs should include relevant rules, internally generated design requirements (unsprung mass target, rotational inertia target etc) and a thorough description of what the designer is required to do.


Once all the design briefs are allocated out, some to a third/fourth year elective (one session long course) and the remainder distributed within the team. As we have a small team design briefs are issued out based on willingness and ability to handle the workload, we loosely have subject matter experts like engine, suspension, electronics. In the past we tried a very hierarchical structure which didn't work because we didn't have enough people to fill all the positions. Designer then conducts JoS which influences the amount of design work that needs to be documented for review and knowledge transfer. From here I would like to hold about four design reviews being:

Conceptual Design Review: Identifying interdependencies with other subsystems, applicable rules, load cases and presenting several possible concepts.

Preliminary Design Review: Present justification of which design should be developed, how you plan to analyse it, how will it be manufactured and who by, how will you verify that you have met design requirements, how does your design support the overall vehicle concept etc.

Detail Design Review: Calculations are reviewed, analysis constraints are justified, evidence of integration is given, method of manufacture and assembly are checked.

Final Design Review: This is to go over any reworks arising out of the Detail Design Review.

Our team structure is flat in that is horizontal with only two team leaders, one administrative and one technical (one is also appointed Team Captain so there is one ‘Chief’). The majority of jobs required to complete the FSAE Project in full are functionally split between administrative and technical. Most team members will hold several positions across the technical and administrative branches.

[oz_olly]

I should aslo add that we are a team in a military university so naturally we have tended towards a multi layer approach that represents the military structure. From what I have seen this type of structure results in a significant management overhead. In our case it has also not supported knowledge transfer between specialisations and it is not very receptive to people who just join in third or fourth year. We need a structure that will accept any level of competence and assign them jobs they are capable of managing.

But not to hijack my own thread. I am more focussed on trying to develop a practically successful design review process and some level of rigour so that we don't have wheels falling off the car or extremely expensive components that don't fit/integrate etc.

[MH]

First of all it's a great thing that you set strict deadlines (I assume the dates and deliverables are known well in time and do not get changed) for the team members, because that more or less forces them to deliver.

However I'd implement something of a requirements analysis review as well. The requirements from the rules of the comp, from your top level goal, from the surroundings etc. are the basis for a well designed car. You might as well have a kick-off with a rules-quiz for all team members. If you have a good set of requirements the verification process is much easier too.

By the time you're doing design reviews, you're correcting mistakes instead of preventing them. I know that the Delft team still does these things (and more) because you can save yourself a lot of trouble by having a good foundation for your design process.

I'm not saying you should do it, I only know it works for some teams. But maybe it's worth a second thought http://fsae.com/groupee_common/emoticons/icon_smile.gif

regards,
Miki Hegedus
Delft University of Technology 2001-2008

[Bemo]

In my opinion the problem about the JOS is that almost every part of a FSAE car has major significance for finishing endurance, passing scrutineering etc. If not you have to ask yourself why the part should be at the car at all.

The most important thing for an efficient design process is that every designer has a well defined task, which he is responsible for. Never just give one task to two people. Try to divide it into two sub-tasks. This makes sure that every designer has a certain work load to carry and results to deliver. Other way round every part of the car must have a responsible.

The rest is definitevly depending on your team size. During my Rennteam-time we usually were around 35 members while only 15 in GreenTeam.
You can be succesful with both team sizes but organisational aspects are very different.

[RobbyObby]

Originally posted by thewoundedsoldier:
Robby,

Would you agree that the segregation of tasks and analysis led to the endurance failure at comp last year?

An issue remained unresolved because of inefficiency in the system. A crack existed and murphy got ya. I only bring up this particular failure because it was a perfect example of subteams not communicating or working well enough together. If that problem was not left to just one guy, maybe other's would have noticed?

I would agree that design review is unquestionably a staple of a successful team. Last year our hybrid team did it monthly, while the FSAE team did it weekly. I kind of robbed the thread and made it about large teams vs. small teams and inefficiency vs. efficiency. Sorry.

Yes, ultimately that failure was caused by a lack of communication. But not because of the team structure. The suspension team was not merely one person but many that ultimately overlooked the problem. However, there is no way that any of our engine guys or composite/body guys would have seen that design and known that rod end was in bending.

But I digress...

This year I am going to strive to implement an overall conceptual design system where key design leads, team leads, and senior project members hold weekly or bi-weekly meetings to discuss major design choices on the car. These include things like 10" or 13" wheel choice, hydraulic shifting or mechanical, naturally aspirated or turbocharged, etc. Once these overall designs are debated and chosen upon by key members, team leads can then present the design to their individual subteams and delegate tasks/responsibilities to other members. This ensures that the key members of your team understand all the design decisions of each subteam, but other members of each subteam aren't expected to know the nitty gritty.