DPDK and energy efficiency

Etienne-Victor Depasquale edepa at ieee.org
Sat Feb 27 16:11:15 UTC 2021


Just a quick note to say that I've closed the survey.

I haven't published the results yet as I said that I would write notes
necessary as a preamble to correctly inform potential readers,
and these notes are taking longer to write than I have time available.

Cheers,

Etienne

On Wed, Feb 24, 2021 at 7:07 PM Etienne-Victor Depasquale <edepa at ieee.org>
wrote:

> I think I need to calm this thread down.
>
> I'm a researcher, and my interest is in the truth, not in my opinion.
>
> I've read some facts in this thread that are necessary
> as a prerequisite to the publication of the results on Friday.
>
> I do want to ensure that no future reader is misinformed and will do my
> best,
> with the help of contribution from my peers in this good community,
> to summarize all objections to this survey's questions,
> in the same message as that which publishes the result.
>
> All peace and good wishes,
>
> Etienne
>
> On Wed, Feb 24, 2021 at 4:35 PM Robert Bays <robert at gdk.org> wrote:
>
>> To the nanog community, I’m sorry to have dragged this conversation out
>> further.  I'm only responding to this because there are a significant
>> number of open source projects and commercial products that use DPDK, or
>> similar userspace network environment in their implementations.  The
>> statements in this thread incorrectly cast them, because they use DPDK, as
>> inefficient.  But the reality is they have all been designed from day one
>> not to unnecessarily consume power.  Please ask your open source dev and/or
>> vendor of choice to verify.  But please don’t rely on the information in
>> this thread to make decisions about what you deploy in your network.
>>
>> On Feb 23, 2021, at 11:44 PM, Etienne-Victor Depasquale <edepa at ieee.org>
>> wrote:
>>
>> Hello Robert,
>>
>> Your statement that DPDK “keeps utilization at 100% regardless of packet
>>> activity” is just not correct.  You further pre-suppose "widespread DPDK's
>>> core operating inefficiency” without any data to backup the operating
>>> inefficacy assertion.
>>>
>>
>> This statement is incorrect.
>> I have provided references (please see earlier e-mails) that investigate
>> the operation of DPDK.
>> These references are items of peer-reviewed research that investigate a
>> perceived problem with deployment of DPDK.
>> If the power consumption incurred while running DPDK were a corner case,
>> then there would be little to no research value in investigating such
>> behavior.
>>
>>
>> Your references don’t take into account the code that this community
>> would actually deploy; open source implementations like DANOS, FD.io, or
>> OVS.  They don’t audit any commercial products that implement userspace
>> stacks.  None of your references say that DPDK is inherently inefficient.
>> The closest they come is to say that tight polling is inefficient.  But
>> tight polling, even in the earliest days of DPDK, was never meant to be a
>> design pattern that was actually deployed into production.  I was there for
>> those early conversations.
>>
>> Please don’t mislead the community into believing that DPDK == power bad
>>>
>> I have to object to this statement. It does seem to imply malice, or, at
>> best, amateurish behaviour, whether you intended it or not.
>>
>>
>> Object all you want.  You are misleading people with your comments.  And
>> in the process you are denigrating a large swath of OSS projects and
>> commercial products that use DPDK.  Your survey questions are leading and
>> provide a false dichotomy.  And when you post the results here, they will
>> be archived forever to continue to spread misinformation, unfortunately.
>>
>> Everything following is informational.  Stop here if so inclined.
>>>
>>  Please stop delving into the detail of DPDK's facilities without regard
>> for your logical omission:
>> that whether the facilities are available or not, DPDK's deployment
>> profile (meaning: how it's being used in general), as indicated by the
>> references I've provided,
>> are leading to high power inefficiency on cores partitioned to the data
>> plane.
>>
>>
>> I’ve been writing network appliance code for over 20 years.  I designed
>> network architectures for years before that.  I have 10s of thousands of
>> DPDK based appliances in production at this moment across multiple
>> different use cases. I work with companies that have 100s of thousands of
>> units in production that leverage userspace runtimes.  I do think I
>> understand DPDK’s deployment profile better than you.  That’s what I have
>> been trying to tell you.  People don’t write inefficient DPDK code to put
>> into production.  We’re not dumb.  We’ve been thinking about power
>> consumption from day one.  DPDK was never supposed to be just a tight loop
>> poll.  You were always supposed to put in the very minimal extra work to
>> modulate power consumption.
>>
>> The takeaway is that DPDK (and similar) doesn’t guarantee runaway power
>>> bills.
>>>
>> Of course it doesn't.
>> Even the second question of that bare-bones survey tried to communicate
>> this much.
>>
>> If you have questions, I’d be happy to discuss off line
>>>
>> I would be happy to answer your objections in detail off line too.
>> Just let me know.
>>
>>
>> Unfortunately, you don’t seem to be receptive to the numerous people
>> contradicting your assertions.  So I’m out.  I’ll let my comments stand
>> here.
>>
>> Cheers,
>>
>> Etienne
>>
>>
>> On Wed, Feb 24, 2021 at 12:12 AM Robert Bays <robert at gdk.org> wrote:
>>
>>> Hi Etienne,
>>>
>>> Your statement that DPDK “keeps utilization at 100% regardless of packet
>>> activity” is just not correct.  You further pre-suppose "widespread DPDK's
>>> core operating inefficiency” without any data to backup the operating
>>> inefficacy assertion.  Your statements, taken at face value, lead people to
>>> believe that if a project uses DPDK it’s going to increase their power
>>> costs.  And that’s just not the case.  Please don’t mislead the community
>>> into believing that DPDK == power bad.
>>>
>>> Everything following is informational.  Stop here if so inclined.
>>>
>>> DPDK does not dictate CPU utilization or power consumption, the
>>> application leveraging DPDK does.  It’s the application that decides how to
>>> poll packets.  If an application implements DPDK using only a tight polling
>>> loop, then it will keep CPU cores that are running DPDK threads at 100%.
>>> But only the most simple and/or bespoke (think trading) applications are
>>> implemented this way.  You don’t need tight polling all the time to get the
>>> performance gains provided by DPDK or similar environments.  The vast
>>> majority of applications that this audience would actually install in their
>>> networks do not do tight polling all the time and therefore don’t consume
>>> 100% of the CPU all the time.   An interesting, helpful research effort you
>>> could lead would be to survey the ecosystem to catalog those applications
>>> that do fall into the power hungry category and help them to change their
>>> code.
>>>
>>> Intel DPDK application development guidelines don’t pre-suppose tight
>>> polling all the time and offer at least two methods for optimizing power
>>> against throughput.  The older method is to use adaptive polling;
>>> increasing the polling frequency as traffic load increases.  This keeps cpu
>>> utilization low when packet load is light and increases it as traffic
>>> levels warrant.  The second method is to use P-states and/or C-states to
>>> put the processor into lower power modes when traffic loads are lighter.
>>> We have found that adaptive polling works better across a larger pool of
>>> hardware types, and therefore that is what DANOS uses, amongst other
>>> things.
>>>
>>> Further, performance and power consumption are dictated by a
>>> multivariate set of application decisions including: design patterns such
>>> as single thread run to completion models vs. passing mbufs between
>>> multiple threads, buffer sizes and cache management algorithms, combining
>>> and/or separating tx/rx threads, binding threads to specific lcores,
>>> reserved cores for DPDK threads, hyperthreading, kernel schedulers,
>>> hypervisor schedulers, interface drivers, etc.  All of these are
>>> application specific, not DPDK generic.  Well written applications that
>>> leverage DPDK provide knobs for the user to tune these settings for their
>>> specific environment and use case.  None of this unique to DPDK.  Solution
>>> designs were cribbed from previous technologies.
>>>
>>> The takeaway is that DPDK (and similar) doesn’t guarantee runaway power
>>> bills.  Power consumption is dictated by the application.  Look for well
>>> behaved applications and everything will be alright.
>>>
>>> If you have questions, I’d be happy to discuss off line.
>>>
>>> Thanks,
>>> Robert.
>>>
>>>
>>> > On Feb 22, 2021, at 11:27 PM, Etienne-Victor Depasquale <
>>> edepa at ieee.org> wrote:
>>> >
>>> > Sorry, last line should have been:
>>> > "intended to get an impression of how widespread ***knowledge of***
>>> DPDK's core operating inefficiency is",
>>> > not:
>>> > "intended to get an impression of how widespread DPDK's core operating
>>> inefficiency is"
>>> >
>>> > On Tue, Feb 23, 2021 at 8:22 AM Etienne-Victor Depasquale <
>>> edepa at ieee.org> wrote:
>>> > Beyond RX/TX CPU affinity, in DANOS you can further tune power
>>> consumption by changing the adaptive polling rate.  It doesn’t, per the
>>> survey, "keep utilization at 100% regardless of packet activity.”
>>> > Robert, you seem to be conflating DPDK
>>> > with DANOS' power control algorithms that modulate DPDK's default
>>> behaviour.
>>> >
>>> > Let me know what you think; otherwise, I'm pretty confident that DPDK
>>> does:
>>> > "keep utilization at 100% regardless of packet activity.”
>>> >
>>> > Keep in mind that this is a bare-bones survey intended for busy,
>>> knowledgeable people (the ones you'd find on NANOG) -
>>> > not a detailed breakdown of modes of operation of DPDK or DANOS.
>>> > DPDK has been designed for fast I/O that's unencumbered by the
>>> trappings of general-purpose OSes,
>>> > and that's the impression that needs to be forefront.
>>> > Power control, as well as any other dimensions of modulation,
>>> > are detailed modes of operation that are well beyond the scope of a
>>> bare-bones 2-question survey
>>> > intended to get an impression of how widespread DPDK's core operating
>>> inefficiency is.
>>> >
>>> > Cheers,
>>> >
>>> > Etienne
>>> >
>>> > On Mon, Feb 22, 2021 at 10:20 PM Robert Bays <robert at gdk.org> wrote:
>>> > Beyond RX/TX CPU affinity, in DANOS you can further tune power
>>> consumption by changing the adaptive polling rate.  It doesn’t, per the
>>> survey, "keep utilization at 100% regardless of packet activity.”  Adaptive
>>> polling changes in DPDK optimize for tradeoffs between power consumption,
>>> latency/jitter and drops during throughput ramp up periods.  Ideally your
>>> DPDK implementation has an algorithm that tries to automatically optimize
>>> based on current traffic patterns.
>>> >
>>> > In DANOS refer to the “system default dataplane power-profile” config
>>> command tree for adaptive polling settings.  Interface RX/TX affinity is
>>> configured on a per interface basis under the “interfaces dataplane” config
>>> command tree.
>>> >
>>> > -robert
>>> >
>>> >
>>> > > On Feb 22, 2021, at 11:46 AM, Jared Geiger <jared at compuwizz.net>
>>> wrote:
>>> > >
>>> > > DANOS lets you specify how many dataplane cores you use versus
>>> control plane cores. So if you put a 16 core host in to handle 2GB of
>>> traffic, you can adjust the dataplane worker cores as needed. Control plane
>>> cores don't stay at 100% utilization.
>>> > >
>>> > > I use that technique plus DANOS runs on VMware (not oversubscribed)
>>> which allows me to use the hardware for other VMs. NICS are attached to the
>>> VM via PCI Passthrough which helps eliminate the overhead to the VMware
>>> hypervisor itself.
>>> > >
>>> > > I have an 8 core VM with 4 cores set to dataplane and 4 to control
>>> plane. The 4 control plane cores are typically idle only processing BGP
>>> route updates, SNMP, logs, etc.
>>> > >
>>> > > ~Jared
>>> > >
>>> > > On Sun, Feb 21, 2021 at 11:30 PM Etienne-Victor Depasquale <
>>> edepa at ieee.org> wrote:
>>> > > Hello folks,
>>> > >
>>> > > I've just followed a thread regarding use of CGNAT and noted a
>>> suggestion (regarding DANOS) that includes use of DPDK.
>>> > >
>>> > > As I'm interested in the breadth of adoption of DPDK, and as I'm a
>>> researcher into energy and power efficiency, I'd love to hear your feedback
>>> on your use of power consumption control by DPDK.
>>> > >
>>> > > I've drawn up a bare-bones, 2-question survey at this link:
>>> > >
>>> > > https://www.surveymonkey.com/r/J886DPY.
>>> > >
>>> > > Responses have been set to anonymous.
>>> > >
>>> > > Cheers,
>>> > >
>>> > > Etienne
>>> > >
>>> > > --
>>> > > Ing. Etienne-Victor Depasquale
>>> > > Assistant Lecturer
>>> > > Department of Communications & Computer Engineering
>>> > > Faculty of Information & Communication Technology
>>> > > University of Malta
>>> > > Web. https://www.um.edu.mt/profile/etiennedepasquale
>>> >
>>> >
>>> >
>>> > --
>>> > Ing. Etienne-Victor Depasquale
>>> > Assistant Lecturer
>>> > Department of Communications & Computer Engineering
>>> > Faculty of Information & Communication Technology
>>> > University of Malta
>>> > Web. https://www.um.edu.mt/profile/etiennedepasquale
>>> >
>>> >
>>> > --
>>> > Ing. Etienne-Victor Depasquale
>>> > Assistant Lecturer
>>> > Department of Communications & Computer Engineering
>>> > Faculty of Information & Communication Technology
>>> > University of Malta
>>> > Web. https://www.um.edu.mt/profile/etiennedepasquale
>>>
>>>
>>
>> --
>> Ing. Etienne-Victor Depasquale
>> Assistant Lecturer
>> Department of Communications & Computer Engineering
>> Faculty of Information & Communication Technology
>> University of Malta
>> Web. https://www.um.edu.mt/profile/etiennedepasquale
>>
>>
>>
>
> --
> Ing. Etienne-Victor Depasquale
> Assistant Lecturer
> Department of Communications & Computer Engineering
> Faculty of Information & Communication Technology
> University of Malta
> Web. https://www.um.edu.mt/profile/etiennedepasquale
>


-- 
Ing. Etienne-Victor Depasquale
Assistant Lecturer
Department of Communications & Computer Engineering
Faculty of Information & Communication Technology
University of Malta
Web. https://www.um.edu.mt/profile/etiennedepasquale
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20210227/20b0eff4/attachment.html>


More information about the NANOG mailing list