bugcrmsh - Bugs: bug #41660, crmsh creates duplicate...

 
 

bug #41660: crmsh creates duplicate name="target-role" value="Stopped" entries

Submitted by:  None
Submitted on:  Fri 21 Feb 2014 05:23:22 PM UTC  
 
Category: NoneSeverity: 3 - Normal
Item Group: NoneStatus: Fixed
Privacy: PublicAssigned to: Kristoffer Grönlund <krig>
Open/Closed: Closed

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Wed 26 Mar 2014 12:17:49 AM UTC, comment #5:

This may be clone related. Here is another clone:

[root@mici-admin ~]# pcs resource disable clusterfs-clone
[root@mici-admin ~]# crm configure show xml clusterfs-clone
[root@mici-admin ~]# crm resource start clusterfs-clone
[root@mici-admin ~]# crm configure show xml clusterfs-clone
[root@mici-admin ~]# pcs resource disable clusterfs-clone
Error: Error performing operation: Invalid argument
Multiple attributes match name=target-role
Value: Started (id=clusterfs-clone-meta-target-role)
Value: Started (id=clusterfs-clone-meta_attributes-target-role)

I have been unable to duplicate with non-cloned resource. pcs does NOT create the second meta_attributes that get created with the clone resource.

E.g.,
<meta_attributes id="clusterfs-clone-meta_attributes">
<nvpair id="clusterfs-clone-meta_attributes-target-role" name="target-role" value="Stopped"/>
</meta_attributes>

I cannot keep from thinking that pcs may not be performing as expected.

Bob Haxo <bobhaxo>
Tue 25 Mar 2014 11:45:28 PM UTC, comment #4:

Kristoffer,

Here is a series of pcs & crm commands that trigger the situation. The order is critical but once it is hit, the only way out seems to be to edit the xml (crm configure edit xml resource).

In short, when a resource is "Started", pcs does not include a value="Started" attribute.

When a resource is "Stopped", pcs creates a new <meta_attributes id="libvirtd-clone-meta_attributes">, with value="Stopped".

If "crm resource start <resource>" in then used to start the resource, two attributes with value='Started' are created.

pcs in turn bombs on this pair of "Started" attributes.

This has been reproducible with 100% of my tests ... all two tests ;-)

Bob Haxo

----------------------------------------------------------------------

[root@mici-admin ~]# pcs resource enable libvirtd-clone
[root@mici-admin ~]# crm configure show xml libvirtd-clone

<meta_attributes id="libvirtd-clone-meta">
<nvpair id="libvirtd-interleave" name="interleave" value="true"/>
<nvpair id="libvirtd-ordered" name="ordered" value="true"/>
</meta_attributes>

[root@mici-admin ~]# pcs resource disable libvirtd-clone
[root@mici-admin ~]# crm configure show xml libvirtd-clone

<meta_attributes id="libvirtd-clone-meta">
<nvpair id="libvirtd-interleave" name="interleave" value="true"/>
<nvpair id="libvirtd-ordered" name="ordered" value="true"/>
</meta_attributes>
<meta_attributes id="libvirtd-clone-meta_attributes">
<nvpair id="libvirtd-clone-meta_attributes-target-role" name="target-role" value="Stopped"/>
</meta_attributes>

[root@mici-admin ~]# crm resource start libvirtd-clone
[root@mici-admin ~]# crm configure show xml libvirtd-clone

<meta_attributes id="libvirtd-clone-meta">
<nvpair id="libvirtd-interleave" name="interleave" value="true"/>
<nvpair id="libvirtd-ordered" name="ordered" value="true"/>
<nvpair id="libvirtd-clone-meta-target-role" name="target-role" value="Started"/>
</meta_attributes>
<meta_attributes id="libvirtd-clone-meta_attributes">
<nvpair id="libvirtd-clone-meta_attributes-target-role" name="target-role" value="Started"/>
</meta_attributes>

[root@mici-admin ~]# pcs resource disable libvirtd-clone
Error: Error performing operation: Invalid argument
Multiple attributes match name=target-role
Value: Started (id=libvirtd-clone-meta-target-role)
Value: Started (id=libvirtd-clone-meta_attributes-target-role)

[root@mici-admin ~]# crm configure show xml libvirtd-clone

<meta_attributes id="libvirtd-clone-meta">
<nvpair id="libvirtd-interleave" name="interleave" value="true"/>
<nvpair id="libvirtd-ordered" name="ordered" value="true"/>
<nvpair id="libvirtd-clone-meta-target-role" name="target-role" value="Started"/>
</meta_attributes>
<meta_attributes id="libvirtd-clone-meta_attributes">
<nvpair id="libvirtd-clone-meta_attributes-target-role" name="target-role" value="Started"/>
</meta_attributes>

----------------------------------------------------------------------

Bob Haxo <bobhaxo>
Tue 25 Mar 2014 09:24:03 PM UTC, comment #3:

[root@mici-admin ~]# crm --version
1.2.6-2.3.1sgi710a22.rhel6bhaxo (Build 4f66cc1901854a7cb60d483414ce5cf2d749db60)
[root@mici-admin ~]# pcs --version
0.9.90

I did again today manage to re-create the issue:

[root@mici-admin ~]# pcs resource enable libvirtd-clone
Error: Error performing operation: Invalid argument
Multiple attributes match name=target-role
Value: Started (id=libvirtd-clone-meta-target-role)
Value: Started (id=libvirtd-clone-meta_attributes-target-role)

I'll try to figure out a more or less reproducible sequence of mixed pcs and crm
commands to generate the duplicate entry:

Extract from "cibadmin --query --scope configuration" :

<meta_attributes id="libvirtd-clone-meta">
<nvpair id="libvirtd-interleave" name="interleave" value="true"/>
<nvpair id="libvirtd-ordered" name="ordered" value="true"/>
<nvpair id="libvirtd-clone-meta-target-role" name="target-role" value="Started"/>
</meta_attributes>
<meta_attributes id="libvirtd-clone-meta_attributes">
<nvpair id="libvirtd-clone-meta_attributes-target-role" name="target-role" value="Started"/>
</meta_attributes>

Bob Haxo <bobhaxo>
Tue 25 Mar 2014 05:13:11 PM UTC, comment #2:

Kristoffer, Thanks for your reply. I'll respond shortly. Bob

Bob Haxo <bobhaxo>
Fri 07 Mar 2014 08:45:19 PM UTC, comment #1:

Hi Bob,

I've been trying to reproduce this and haven't gotten things to behave this badly. These are the versions I've tried:

s12-one:~ # crm --version
2.0-33.1 (Build dad87af7e64ba415b13edce29dceb53747139720)
s12-one:~ # pcs --version
0.9.91

What version of crmsh are you using?

Also, if this is something you can reproduce reliably, I would be grateful for a step-by-step process on how to reproduce it. It would also be helpful to get the debug output from using crm -d while doing this.

Thanks!
Kristoffer

Kristoffer Grönlund <krig>
Project AdministratorIn charge of this item.
Fri 21 Feb 2014 05:23:22 PM UTC, original submission:

From -unavailable- Thu Feb 20 16:03:37 2014
Subject: Re: [Pacemaker] possible regex error in "pcs resource enable/disable"
From: Bob Haxo <bhaxo@sgi.com>
To: Andrew Beekhof <andrew@beekhof.net>
Cc: The Pacemaker cluster resource manager <pacemaker@oss.clusterlabs.org>, Lars Marowsky-Bree <lmb@suse.com>

On Fri, 2014-02-21 at 10:46 +1100, Andrew Beekhof wrote:

> On 21 Feb 2014, at 10:25 am, Bob Haxo <bhaxo@sgi.com> wrote:
>
> > Andrew,
> >
> >> From my vantage point, it will be most unfortunate if pcs and crm cannot
> > both be used for routine start and stop of resources, and other routine
> > tasks.
>
> Agreed.
> AFAICS, it's crmsh thats creating the duplicate entries though.
> Not much pcs can do about that.


Yes, it is the crm that creates the duplicate entries. I have included
below the effect of pcs disable+enable and crm stop on the xml.

Sooo, seems that we need to kick this over the fence to the crmsh
folks ... or the SUSE folks, if they are maintaining crmsh.

------------------------------
mixing crm and pcs cmds
------------------------------

[root@mici-admin ~]# pcs resource disable libvirtd-clone
[root@mici-admin ~]# cibadmin --query --scope configuration > cib-config-pcs-disable-20140220-0855.xml
[root@mici-admin ~]# diff -u cib-config-initial-enabled-20140220-0854.xml cib-config-pcs-disable-20140220-0855.xml
--- cib-config-initial-enabled-20140220-0854.xml 2014-02-20 10:54:18.000000000 -0600
+++ cib-config-pcs-disable-20140220-0855.xml 2014-02-20 10:54:50.000000000 -0600
@@ -74,7 +74,9 @@
<nvpair id="libvirtd-interleave" name="interleave" value="true"/>
<nvpair id="libvirtd-ordered" name="ordered" value="true"/>
</meta_attributes>
- <meta_attributes id="libvirtd-clone-meta_attributes"/>
+ <meta_attributes id="libvirtd-clone-meta_attributes">
+ <nvpair id="libvirtd-clone-meta_attributes-target-role" name="target-role" value="Stopped"/>
+ </meta_attributes>
</clone>
<primitive class="ocf" id="virt" provider="heartbeat" type="VirtualDomain">
<instance_attributes id="virt-instance_attributes">

[root@mici-admin ~]# pcs resource enable libvirtd-clone
[root@mici-admin ~]# cibadmin --query --scope configuration > cib-config-pcs-enable-20140220-0900.xml

[root@mici-admin ~]# diff cib-config-initial-enabled-20140220-0854.xml cib-config-pcs-enable-20140220-0900.xml
[root@mici-admin ~]# <no diff>

[root@mici-admin ~]# crm resource stop libvirtd-clone
[root@mici-admin ~]# cibadmin --query --scope configuration > cib-config-crm-stop-20140220-0904.xml

[root@mici-admin ~]# diff -u cib-config-initial-enabled-20140220-0854.xml cib-config-crm-stop-20140220-0904.xml
--- cib-config-initial-enabled-20140220-0854.xml 2014-02-20 10:54:18.000000000 -0600
+++ cib-config-crm-stop-20140220-0904.xml 2014-02-20 11:04:34.000000000 -0600
@@ -73,8 +73,11 @@
<meta_attributes id="libvirtd-clone-meta">
<nvpair id="libvirtd-interleave" name="interleave" value="true"/>
<nvpair id="libvirtd-ordered" name="ordered" value="true"/>
+ <nvpair id="libvirtd-clone-meta-target-role" name="target-role" value="Stopped"/>
+ </meta_attributes>
+ <meta_attributes id="libvirtd-clone-meta_attributes">
+ <nvpair id="libvirtd-clone-meta_attributes-target-role" name="target-role" value="Stopped"/>
</meta_attributes>
- <meta_attributes id="libvirtd-clone-meta_attributes"/>
</clone>
<primitive class="ocf" id="virt" provider="heartbeat" type="VirtualDomain">
<instance_attributes id="virt-instance_attributes">

Bob Haxo

>
> >
> > I am using pcs (almost exclusively) for my RHEL6.5 Pacemaker port.
> >
> > But, I am *REALLY* hoping that I can present a relatively consistent
> > crmsh interface to the consumers of my HA work (customers, in-house
> > engineers) with RHEL and SLES installations. At least the in-house
> > engineers find the HA very confusing. This sort of a break will make it
> > impossible to ship with crmsh, thus continuing the confusion of two
> > distro specific interfaces.
> >
> > Regards,
> > Bob Haxo
> >
> >
> > On Fri, 2014-02-21 at 09:26 +1100, Andrew Beekhof wrote:
> >> On 21 Feb 2014, at 4:30 am, Bob Haxo <bhaxo@sgi.com> wrote:
> >>
> >>> Andrew, Lars,
> >>>
> >>> Yes, I have determined that this error is the result of mixing crmsh
> >> stop/start with pcs disable/enable (or maybe pcs stop/start mis-usage) commands.
> >>
> >> Specifically it will be when you use pcs first and crmsh afterwards.
> >> I just checked, pcs (which uses crm_resource) does the right thing if
> >> either one exists (ie. doesn't create a second entry), but necessarily
> >> bails out if both do.
> >>
> >> (libvirtd-clone-meta_attributes is the one created by crm_resource/pcs)
> >>
> >>>
> >>> I've started to respond with how this happens, but have been pulled
> >> off to a higher priority task. Back when I have the resolved.
> >>>
> >>> Bob Haxo
> >>>
> >>> On Thu, 2014-02-20 at 07:56 -0800, Bob Haxo wrote:
> >>>> Andrew,
> >>>>
> >>>> I think that this is what you identified:
> >>>>
> >>>> This error appears to be related to the two instances of <meta_attributes> with
> >>>> libvirtd-clone, both which contain a name="target-role" value="Stopped".
> >>>> Deleted one of these entries and the error did not occur.
> >>>>
> >>>> <clone id="libvirtd-clone">
> >>>> <primitive class="lsb" id="libvirtd" type="libvirtd">
> >>>> <operations>
> >>>> <op id="libvirtd-monitor-interval-15" interval="15" name="monitor" start-delay="15" timeout="15"/>
> >>>> <op id="libvirtd-start-interval-0" interval="0" name="start" on-fail="restart" timeout="15"/>
> >>>> <op id="libvirtd-stop-interval-0" interval="0" name="stop" on-fail="ignore" timeout="15"/>
> >>>> </operations>
> >>>> <meta_attributes id="libvirtd-meta_attributes"/>
> >>>> </primitive>
> >>>> <meta_attributes id="libvirtd-clone-meta">
> >>>> <nvpair id="libvirtd-interleave" name="interleave" value="true"/>
> >>>> <nvpair id="libvirtd-ordered" name="ordered" value="true"/>
> >>>> <nvpair id="libvirtd-clone-meta-target-role" name="target-role" value="Stopped"/>
> >>>> </meta_attributes>
> >>>> <meta_attributes id="libvirtd-clone-meta_attributes">
> >>>> <nvpair id="libvirtd-clone-meta_attributes-target-role" name="target-role" value="Stopped"/>
> >>>> </meta_attributes>
> >>>> </clone>
> >>>>
> >>>> I do occasionally use a crmsh command, mostly the "crm resource cleanup <resource>" ...
> >>>> the pcs version of this seems to never cleanup the remote system entries. I classify
> >>>> this as a bug that I have not reported.
> >>>>
> >>>> So, yes, how the duplicate entry got there is probably the crux of the issue. And
> >>>> I have no answer. I have not used crmsh to create resources (the creates are
> >>>> scripted using pcs cmds). Perhaps I used a "crm stop" rather than a "pcs disable".
> >>>>
> >>>> Regardless, even if RH is not going to support crmsh, at least RH needs to be sensitive
> >>>> to the use of crmsh with their PCS Pacemaker implementation.
> >>>>
> >>>> I'll test/investigate the "crm stop" and see whether this creates the additional entry.
> >>>>
> >>>> Cheers,
> >>>> Bob Haxo
> >>>>
> >>>> On Thu, 2014-02-20 at 10:12 +1100, Andrew Beekhof wrote:
> >>>>> Have you been mixing pcs and crmsh again? :-)
> >>>>> The interesting part is how the dup got in there in the first place.
> >>>>>
> >>>>> Can you remove both settings and try to recreate that step?
> >>>>>
> >>>>> On 20 Feb 2014, at 9:39 am, Bob Haxo <bhaxo@sgi.com> wrote:
> >>>>>
> >>>>>> Chris, was easy to duplicate ... I thought that I had cleared
> >>>>>> the error, but that had not happened.
> >>>>>>
> >>>>>> Bob Haxo
> >>>>>>
> >>>>>> [root@mici-admin ~]# pcs resource disable virt
> >>>>>> [root@mici-admin ~]# pcs resource disable libvirtd-clone
> >>>>>> Error: Error performing operation: Invalid argument
> >>>>>> Multiple attributes match name=target-role
> >>>>>> Value: Started (id=libvirtd-clone-meta-target-role)
> >>>>>> Value: Started (id=libvirtd-clone-meta_attributes-target-role)
> >>>>>>
> >>>>>> [root@mici-admin ~]# pcs --debug resource disable libvirtd-clone
> >>>>>> Running: /usr/sbin/crm_resource -r libvirtd-clone -m -p target-role -v Stopped
> >>>>>> Return Value: 22
> >>>>>> --Debug Output Start--
> >>>>>> Error performing operation: Invalid argument
> >>>>>> Multiple attributes match name=target-role
> >>>>>> Value: Started (id=libvirtd-clone-meta-target-role)
> >>>>>> Value: Started (id=libvirtd-clone-meta_attributes-target-role)
> >>>>>>
> >>>>>> --Debug Output End--
> >>>>>>
> >>>>>> Error: Error performing operation: Invalid argument
> >>>>>> Multiple attributes match name=target-role
> >>>>>> Value: Started (id=libvirtd-clone-meta-target-role)
> >>>>>> Value: Started (id=libvirtd-clone-meta_attributes-target-role)
> >>>>>>
> >>>>>> [root@mici-admin ~]# pcs resource disable libvirtd-clone
> >>>>>> Error: Error performing operation: Invalid argument
> >>>>>> Multiple attributes match name=target-role
> >>>>>> Value: Started (id=libvirtd-clone-meta-target-role)
> >>>>>> Value: Started (id=libvirtd-clone-meta_attributes-target-role)
> >>>>>>
> >>>>>>
> >>>>>> On Wed, 2014-02-19 at 15:23 -0600, Chris Feist wrote:
> >>>>>>> On 02/19/2014 11:16 AM, Bob Haxo wrote:
> >>>>>>>> Encountered this error with pcs but not with crm. Looks like a regex
> >>>>>>>> error, with the existing regex grabbing all strings starting with the
> >>>>>>>> string "libvirtd-clone" ...
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> [root@mici-admin2 ~]# pcs resource disable libvirtd-clone
> >>>>>>>> Error: Error performing operation: Invalid argument
> >>>>>>>> Multiple attributes match name=target-role
> >>>>>>>> Value: Started (id=libvirtd-clone-meta-target-role)
> >>>>>>>> Value: Started (id=libvirtd-clone-meta_attributes-target-role)
> >>>>>>>>
> >>>>>>>> [root@mici-admin ~]# crm resource stop libvirtd-clone
> >>>>>>>>
> >>>>>>>> <<no error, success>>
> >>>>>>>>
> >>>>>>>> [root@mici-admin ~]# pcs resource enable libvirtd-clone
> >>>>>>>> Error: Error performing operation: Invalid argument
> >>>>>>>> Multiple attributes match name=target-role
> >>>>>>>> Value: Stopped (id=libvirtd-clone-meta-target-role)
> >>>>>>>> Value: Stopped (id=libvirtd-clone-meta_attributes-target-role)
> >>>>>>>>
> >>>>>>>> xml:
> >>>>>>>> <clone id="libvirtd-clone">
> >>>>>>>> <meta_attributes id="libvirtd-clone-meta">
> >>>>>>>> <nvpair id="libvirtd-clone-meta-target-role" name="target-role" value="Stopped"/>
> >>>>>>>> <meta_attributes id="libvirtd-clone-meta_attributes">
> >>>>>>>> <nvpair id="libvirtd-clone-meta_attributes-target-role" name="target-role" value="Stopped"/>
> >>>>>>>>
> >>>>>>>> Regards,
> >>>>>>>> Bob Haxo
> >>>>>>>
> >>>>>>> Can you send the output of the following command:
> >>>>>>> pcs --debug resource disable libvirtd-clone
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Chris
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> _____________________________________________
> >>>>>>>> Pacemaker mailing list: -unavailable-
> >>>>>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> >>>>>>>>
> >>>>>>>> Project Home: http://www.clusterlabs.org
> >>>>>>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> >>>>>>>> Bugs: http://bugs.clusterlabs.org

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by bobhaxo (Posted a comment)
  • -unavailable- added by krig (Updated the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 5 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue 08 Apr 2014 01:21:22 PM UTCkrigOpen/ClosedOpen=>Closed
    Tue 08 Apr 2014 01:21:14 PM UTCkrigStatusNeed Info=>Fixed
    Fri 07 Mar 2014 08:45:33 PM UTCkrigStatusConfirmed=>Need Info
    Fri 21 Feb 2014 07:02:31 PM UTCkrigStatusNone=>Confirmed
      Assigned toNone=>krig

    Back to the top


    Powered by Savane 3.1-cleanup1